diff --git a/.gitignore b/.gitignore index c0de440f..a31a8e6c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,4 @@ -*/.vscode +*.vscode *.classpath *.project *.prefs diff --git a/README.md b/README.md index 3bbd4105..a08cf9e1 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,8 @@ KSAN은 AWS S3 호환 API를 기본적으로 제공하고 개발 로드맵에 ℹ️ Microsoft Azure API 및 Google Cloud API는 각각 2022년, 2023년에 단계적으로 지원할 예정입니다. +ℹ️ Microsoft Azure API는 v1.2.0부터 지원합니다. +

### 미션 크리티컬 서비스에 즉시 도입 가능한 고가용성을 지원 @@ -52,9 +54,9 @@ KSAN 시스템은 메타데이터를 관리하기 위해 MariaDB와 같은 RDBMS 그리고 KSAN 시스템의 오브젝트 데이터는 서로 다른 OSD의 OSDDISK에 복제본을 배치 및 저장하는 방식으로 단일 지점의 물리적인 장애에 대응하도록 설계되었습니다. 또한 ksanGW는 HAProxy 등의 서비스 로드밸런서를 이용해 오브젝트 스토리지 서비스의 가용성을 보장할 수 있습니다. -또한 KSAN 시스템은 재해 상황에 대해서도 서비스 가용성을 보장할 수 있습니다. 2022년 3분기 이후에 공개 및 제공이 예정되어 있는 ksanDR 모듈은 지역적으로 배치되는 KSAN 시스템들 간의 오브젝트 데이터를 실시간으로 동기화하는 기능을 제공할 예정입니다. +또한 KSAN 시스템은 재해 상황에 대해서도 서비스 가용성을 보장할 수 있습니다. ksanReplicationManager 모듈은 ksanGW에서 기록하는 Service Log를 기반으로 다른 지역에 구축된 KSAN 시스템들 간의 오브젝트 데이터를 실시간으로 동기화하는 기능을 제공합니다. -ℹ️ 재해 상황에 대한 서비스 고가용성 지원을 위한 ksanDR 기능은 2022년 4분기에 공개될 예정입니다. +ℹ️ 재해 상황에 대한 서비스 고가용성 지원을 위한 ksanReplicationManager 기능은 v1.2.0 부터 지원합니다.

@@ -64,7 +66,7 @@ KSAN 시스템은 메타데이터를 관리하기 위해 MariaDB와 같은 RDBMS 또한 1+1 복제 방식으로 OSDDISK에 분산 저장된 오브젝트 데이터를 특정 시간이 지나면 자동으로 Erasure Coding으로 처리해 더 적은 백엔드 저장 자원으로 오브젝트 데이터를 보관할 수 있도록 설계되었습니다. -ℹ️ KSAN의 Erasure Coding 기능은 2022년 4분기에 공개될 예정입니다. +ℹ️ KSAN의 Erasure Coding 기능은 v1.2.0 부터 지원합니다.

diff --git a/core/backend/lifecycle/DockerFile b/core/backend/lifecycle/DockerFile index 173baad0..0ab35439 100644 --- a/core/backend/lifecycle/DockerFile +++ b/core/backend/lifecycle/DockerFile @@ -18,6 +18,6 @@ FROM openjdk:11-jre-slim WORKDIR /app # RUN mkdir -p /usr/local/ksan/etc/ # COPY ./core/backend/lifecycle/resources/ksanLifecycle_log_conf.xml /usr/local/ksan/etc/ -COPY --from=build /src/target/ksanLifecycle.jar /usr/local/ksan/sbin/ +COPY --from=build /src/target/ksanLifecycleManager.jar /usr/local/ksan/sbin/ COPY ./core/backend/lifecycle/resources/start.sh /app CMD ["/app/start.sh"] \ No newline at end of file diff --git a/core/backend/lifecycle/README.md b/core/backend/lifecycle/README.md index 1acf06ce..3f0eaeb6 100644 --- a/core/backend/lifecycle/README.md +++ b/core/backend/lifecycle/README.md @@ -64,12 +64,12 @@ systemctl start docker ### Build -- 실행후 ksanLifecycle.tar 이미지 파일이 생성됩니다. +- 실행후 ksanLifecycleManager.tar 이미지 파일이 생성됩니다. ```shell #!/bin/bash -docker build --rm -t pspace/ksan-lifecycle:latest -f DockerFile . -docker save -o ksanLifecycle.tar pspace/ksan-lifecycle +docker build --rm -t pspace/ksan-lifecycle-manager:latest -f DockerFile . +docker save -o ksanLifecycleManager.tar pspace/ksan-lifecycle-manager ``` ## How to Use @@ -102,8 +102,8 @@ docker save -o ksanLifecycle.tar pspace/ksan-lifecycle #### 로그 설정 -- 파일명 : `ksan-lifecycle.xml` -- 경로 : `/app/ksan-lifecycle.xml` +- 파일명 : `ksan-lifecycle-manager_log_conf.xml` +- 경로 : `/app/ksan-lifecycle-manager_log_conf.xml` ```xml @@ -133,7 +133,7 @@ docker save -o ksanLifecycle.tar pspace/ksan-lifecycle #### 프로그램 설치 ``` shell # 빌드 했을 경우 이미지 로드 -docker load -i /root/docker/ksanLifecycle.tar +docker load -i /root/docker/ksanLifecycleManager.tar # 컨테니어 생성 docker create -i -t \ @@ -141,12 +141,12 @@ docker create -i -t \ -v /etc/localtime:/etc/localtime:ro \ -v /var/log/ksan:/app/logs \ -v /usr/local/ksan/etc:/usr/local/ksan \ ---name ksan-lifecycle \ -pspace/ksan-lifecycle:latest +--name ksan-lifecycle-manager \ +pspace/ksan-lifecycle-manager:latest ``` #### 실행예시(CLI) ```bash -docker start ksan-lifecycle +docker start ksan-lifecycle-manager ``` diff --git a/core/backend/lifecycle/pom.xml b/core/backend/lifecycle/pom.xml index af4f7222..d38b2b84 100644 --- a/core/backend/lifecycle/pom.xml +++ b/core/backend/lifecycle/pom.xml @@ -1,8 +1,8 @@ 4.0.0 - KsanLifecycle - ksan-lifecycle + KsanLifecycleManager + ksan-lifecycle-manager 1.0.0 @@ -146,7 +146,7 @@ really-executable-jar-maven-plugin 1.4.1 - ksanLifecycle.jar + ksanLifecycleManager.jar diff --git a/core/backend/lifecycle/resources/ksanLifecycle_log_conf.xml b/core/backend/lifecycle/resources/ksanLifecyclemanager_log_conf.xml similarity index 84% rename from core/backend/lifecycle/resources/ksanLifecycle_log_conf.xml rename to core/backend/lifecycle/resources/ksanLifecyclemanager_log_conf.xml index 5b4ee41c..b3627b7e 100644 --- a/core/backend/lifecycle/resources/ksanLifecycle_log_conf.xml +++ b/core/backend/lifecycle/resources/ksanLifecyclemanager_log_conf.xml @@ -1,13 +1,13 @@ - + - ${LIFECYCLE_LOGS_PATH}/lifecycle.log + ${LOGS_PATH}/lifecycle.log - ${LIFECYCLE_LOGS_PATH}/lifecycle.log.%d{yyyy-MM-dd, ${logback.timezone:-Asia/Seoul}}.%i.log.gz + ${LOGS_PATH}/lifecycle.log.%d{yyyy-MM-dd, ${logback.timezone:-Asia/Seoul}}.%i.log.gz 7 100MB diff --git a/core/backend/lifecycle/resources/start.sh b/core/backend/lifecycle/resources/start.sh index 6b54fd29..c025487c 100644 --- a/core/backend/lifecycle/resources/start.sh +++ b/core/backend/lifecycle/resources/start.sh @@ -1,2 +1,2 @@ #!/bin/bash -java -jar -Dlogback.configurationFile=/usr/local/ksan/etc/ksanLifecycle_log_conf.xml /usr/local/ksan/sbin/ksanLifecycle.jar \ No newline at end of file +java -jar -Dlogback.configurationFile=/usr/local/ksan/etc/ksanLifecycleManager_log_conf.xml /usr/local/ksan/sbin/ksanLifecycleManager.jar \ No newline at end of file diff --git a/core/backend/lifecycle/setup.py b/core/backend/lifecycle/setup.py index 218cc87d..c8537a0b 100644 --- a/core/backend/lifecycle/setup.py +++ b/core/backend/lifecycle/setup.py @@ -16,6 +16,6 @@ setup( name='ksan-lifecycle-util', version=Version, - data_files=[('/usr/local/ksan/etc', ['./ksanLifecycle_log_conf.xml'])], + data_files=[('/usr/local/ksan/etc', ['./ksanLifecycleManager_log_conf.xml'])], ) diff --git a/core/backend/lifecycle/src/com/pspace/Ksan/PortalManager.java b/core/backend/lifecycle/src/com/pspace/Ksan/PortalManager.java index 279e3d17..6e66e989 100644 --- a/core/backend/lifecycle/src/com/pspace/Ksan/PortalManager.java +++ b/core/backend/lifecycle/src/com/pspace/Ksan/PortalManager.java @@ -94,7 +94,7 @@ private String GetURL() { } private String GetConfigURL() { - return String.format("%s/api/v1/Config/KsanLifecycle", GetURL()); + return String.format("%s/api/v1/Config/KsanLifecycleManager", GetURL()); } private String GetRegionURL(String RegionName) { diff --git a/core/backend/lifecycle/src/com/pspace/backend/Data/LifecycleConstants.java b/core/backend/lifecycle/src/com/pspace/backend/Data/LifecycleConstants.java index fec596ab..63d0f07f 100644 --- a/core/backend/lifecycle/src/com/pspace/backend/Data/LifecycleConstants.java +++ b/core/backend/lifecycle/src/com/pspace/backend/Data/LifecycleConstants.java @@ -2,7 +2,7 @@ public class LifecycleConstants { public static final String AGENT_CONF_PATH = "/usr/local/ksan/etc/ksanAgent.conf"; - public static final String LIFECYCLE_SERVICE_ID_PATH = "/usr/local/ksan/sbin/.ksanLifecycle.ServiceId"; + public static final String LIFECYCLE_SERVICE_ID_PATH = "/usr/local/ksan/sbin/.ksanLifecycleManager.ServiceId"; public static final String MQ_EXCHANGE_NAME = "ksan.system"; public static final String MQ_BINDING_KEY = "*.services.state"; } diff --git a/core/backend/metering/README.md b/core/backend/metering/README.md index 1cf5b0ad..17d9a153 100644 --- a/core/backend/metering/README.md +++ b/core/backend/metering/README.md @@ -87,7 +87,7 @@ docker save -o ksanMetering.tar infinistor/ksan-metering #### 프로그램 설치 ``` shell # 빌드 했을 경우 이미지 로드 -docker load -i /root/docker/ksanLifecycle.tar +docker load -i /root/docker/ksanLifecycleManager.tar # 컨테니어 생성 docker create -i -t \ diff --git a/core/backend/setup.py b/core/backend/setup.py deleted file mode 100755 index 52a2443d..00000000 --- a/core/backend/setup.py +++ /dev/null @@ -1,23 +0,0 @@ -#!/usr/bin/python3 - -import sys -import os -from glob import glob -from os.path import basename, splitext -from setuptools import find_packages, setup -import site -import pdb - -UserSitePackagePath = site.getusersitepackages() - -Version = '0.8.0' - - -setup( - name='ksan-mgs-util', - version=Version, - data_files=[('/usr/local/ksan/sbin', [\ - './recovery/ksanRecovery', 'ksanRecovery.jar'])], -) -os.system("chmod 755 /usr/local/ksan/sbin/*") - diff --git a/core/backend/src/com/pspace/backend/libs/Config/LifecycleConfig.java b/core/backend/src/com/pspace/backend/libs/Config/LifecycleConfig.java new file mode 100644 index 00000000..dcb4cd03 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Config/LifecycleConfig.java @@ -0,0 +1,45 @@ +package com.pspace.backend.libs.Config; + +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class LifecycleConfig { + + @JsonProperty("objM.db_repository") + public String DBType; + + @JsonProperty("objM.db_host") + public String Host; + + @JsonProperty("objM.db_port") + public int Port; + + @JsonProperty("objM.db_name") + public String DatabaseName; + + @JsonProperty("objM.db_user") + public String User; + + @JsonProperty("objM.db_password") + public String Password; + + @JsonProperty("ksan.region") + public String Region; + + @JsonProperty("lifecycle.schedule") + public String Schedule; + + @JsonProperty("lifecycle.check_interval") + public long CheckInterval; + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Config/LogManagerConfig.java b/core/backend/src/com/pspace/backend/libs/Config/LogManagerConfig.java new file mode 100644 index 00000000..b639501d --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Config/LogManagerConfig.java @@ -0,0 +1,64 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Config; + +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class LogManagerConfig { + + @JsonProperty("ksan.region") + public String Region; + + @JsonProperty("logM.db_repository") + public String DBType; + + @JsonProperty("logM.db_host") + public String DBHost; + + @JsonProperty("logM.db_port") + public int DBPort; + + @JsonProperty("logM.db_name") + public String DBName; + + @JsonProperty("logM.db_user") + public String DBUser; + + @JsonProperty("logM.db_password") + public String DBPassword; + + @JsonProperty("logM.db_pool_size") + public int DBPoolSize; + + @JsonProperty("logM.db_expires") + public int DBExpires; + + @JsonProperty("logM.check_interval") + public int CheckInterval; + + @JsonProperty("logM.meter_minute") + public int MeterMinute; + + @JsonProperty("logM.assert_hour") + public int AssertHour; + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Config/ReplicationConfig.java b/core/backend/src/com/pspace/backend/libs/Config/ReplicationConfig.java new file mode 100644 index 00000000..b615ed74 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Config/ReplicationConfig.java @@ -0,0 +1,55 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Config; + +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class ReplicationConfig { + + @JsonProperty("objM.db_repository") + public String DBType; + + @JsonProperty("objM.db_host") + public String DBHost; + + @JsonProperty("objM.db_port") + public int DBPort; + + @JsonProperty("objM.db_name") + public String DBName; + + @JsonProperty("objM.db_user") + public String DBUser; + + @JsonProperty("objM.db_password") + public String DBPassword; + + @JsonProperty("ksan.region") + public String Region; + + @JsonProperty("replication.upload_thread_count") + public int ReplicationUploadThreadCount; + + @JsonProperty("replication.multipart_size") + public long ReplicationPartSize; + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/BackendHeaders.java b/core/backend/src/com/pspace/backend/libs/Data/BackendHeaders.java new file mode 100644 index 00000000..35c11dbe --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/BackendHeaders.java @@ -0,0 +1,27 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data; + +public class BackendHeaders { + public static final String S3_NOT_ACTIVATED = "This S3 is not active."; + + public static final String HEADER_DATA = "NONE"; + public static final String HEADER_BACKEND = "x-ifs-admin"; + + public static final String HEADER_REPLICATION = "x-ifs-replication"; + public static final String HEADER_VERSIONID = "x-ifs-version-id"; + + public static final String HEADER_LOGGING = "x-ifs-logging"; + public static final String HEADER_LIFECYCLE = "x-ifs-lifecycle"; + + public static final String S3PROXY_HEADER_NO_DR = "x-amz-ifs-nodr"; + public static final String S3PROXY_HEADER_VERSIONID = "x-amz-ifs-VersionId"; +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/Constants.java b/core/backend/src/com/pspace/backend/libs/Data/Constants.java new file mode 100644 index 00000000..88eb279e --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/Constants.java @@ -0,0 +1,53 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data; + +public class Constants { + public static final String AGENT_CONF_PATH = "/usr/local/ksan/etc/ksanAgent.conf"; + public static final String LIFECYCLE_SERVICE_ID_PATH = "/usr/local/ksan/sbin/.ksanLifecycleManager.ServiceId"; + public static final String LOGMANAGER_SERVICE_ID_PATH = "/usr/local/ksan/sbin/.ksanLogManager.ServiceId"; + public static final String REPLICATION_SERVICE_ID_PATH = "/usr/local/ksan/sbin/.ksanReplicationManager.ServiceId"; + + public static final String MQ_KSAN_SYSTEM_EXCHANGE = "ksan.system"; + public static final String MQ_KSAN_LOG_EXCHANGE = "ksan.log"; + public static final String MQ_HEARTBEAT_BINDING_KEY = "*.services.state"; + + public static final String MQ_QUEUE_LIFECYCLE_EVENT_ADD = "ksan-log-manager-lifecycle-event-add"; + + public static final String MQ_QUEUE_REPLICATION_S3_LOG = "ksan-replication-s3-log"; + public static final String MQ_QUEUE_REPLICATION_EVENT_ADD = "ksan-replication-event-add"; + + public static final String MQ_QUEUE_LOG_MANAGER_S3_LOG = "ksan-log-manager-s3-log"; + public static final String MQ_QUEUE_LOG_MANAGER_LIFECYCLE_EVENT_LOG = "ksan-log-manager-lifecycle-event-log"; + public static final String MQ_QUEUE_LOG_MANAGER_REPLICATION_EVENT_LOG = "ksan-log-manager-replication-event-log"; + + + public static final String MQ_BINDING_GW_LOG = "*.services.gw.log.add"; + public static final String MQ_BINDING_REPLICATION_LOG = "*.services.replication.log.add"; + public static final String MQ_BINDING_REPLICATION_EVENT = "*.services.replication.event.add"; + + public static final String MQ_BINDING_LIFECYCLE_LOG = "*.services.lifecycle.log.add"; + public static final String MQ_BINDING_LIFECYCLE_EVENT = "*.services.lifecycle.event.add"; + + public static final String MQ_EXCHANGE_OPTION_DIRECT = "direct"; + public static final String MQ_EXCHANGE_OPTION_FANOUT = "fanout"; + public static final String MQ_EXCHANGE_OPTION_TOPIC = "topic"; + + public static final String EM_S3_NOT_WORKING = "This S3 does not work!"; + + public static final String DB_TYPE_MARIADB = "MariaDB"; + public static final String DB_TYPE_MONGODB = "MongoDB"; + + public static final String MQ_QUEUE_NAME(String QueueName, String ServiceId) + { + return QueueName + "-" + ServiceId; + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/Lifecycle/LifecycleEventData.java b/core/backend/src/com/pspace/backend/libs/Data/Lifecycle/LifecycleEventData.java new file mode 100644 index 00000000..3fc16d0f --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/Lifecycle/LifecycleEventData.java @@ -0,0 +1,58 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data.Lifecycle; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class LifecycleEventData { + + public String BucketName; + public String ObjectName; + public String VersionId; + public String UploadId; + + public LifecycleEventData() { + Init(); + } + + public LifecycleEventData(LifecycleEventData data) { + this.BucketName = data.BucketName; + this.ObjectName = data.ObjectName; + this.VersionId = data.VersionId; + this.UploadId = data.UploadId; + } + + public LifecycleEventData(String bucketName, String objectName, String versionId, String uploadId) { + this.BucketName = bucketName; + this.ObjectName = objectName; + this.VersionId = versionId; + this.UploadId = uploadId; + } + + public void Init() + { + this.BucketName = ""; + this.ObjectName = ""; + this.VersionId = ""; + this.UploadId = ""; + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/Lifecycle/LifecycleLogData.java b/core/backend/src/com/pspace/backend/libs/Data/Lifecycle/LifecycleLogData.java new file mode 100644 index 00000000..8fba25c8 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/Lifecycle/LifecycleLogData.java @@ -0,0 +1,58 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data.Lifecycle; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.pspace.backend.libs.Utility; + +public class LifecycleLogData extends LifecycleEventData { + + public String Date; + public String Message; + + public LifecycleLogData() { + Init(); + } + + public LifecycleLogData(LifecycleEventData data, String message) { + super(data); + this.Date = Utility.GetNowTime(); + this.Message = message; + } + + public LifecycleLogData(String bucketName, String objectName, String date, String versionId, String uploadId, + String message) { + super(bucketName, objectName, versionId, uploadId); + this.Date = date; + this.Message = message; + } + + @Override + public void Init() { + this.BucketName = ""; + this.ObjectName = ""; + this.VersionId = ""; + this.UploadId = ""; + this.Date = ""; + this.Message = ""; + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/Replication/ReplicationEventData.java b/core/backend/src/com/pspace/backend/libs/Data/Replication/ReplicationEventData.java new file mode 100644 index 00000000..23633e28 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/Replication/ReplicationEventData.java @@ -0,0 +1,66 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data.Replication; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class ReplicationEventData { + + public String Operation; + public String ObjectName; + public String VersionId; + public String SourceBucketName; + public String TargetBucketName; + public String TargetRegion; + + public ReplicationEventData() { + Init(); + } + + public ReplicationEventData(ReplicationEventData data) { + this.Operation = data.Operation; + this.ObjectName = data.ObjectName; + this.VersionId = data.VersionId; + this.SourceBucketName = data.SourceBucketName; + this.TargetBucketName = data.TargetBucketName; + this.TargetRegion = data.TargetRegion; + } + + public ReplicationEventData(String Operation, String ObjectName, String VersionId, String SourceBucketName, + String TargetBucketName, String TargetRegion) { + this.Operation = Operation; + this.ObjectName = ObjectName; + this.VersionId = VersionId; + this.SourceBucketName = SourceBucketName; + this.TargetBucketName = TargetBucketName; + this.TargetRegion = TargetRegion; + } + + public void Init() { + this.Operation = ""; + this.ObjectName = ""; + this.VersionId = ""; + this.SourceBucketName = ""; + this.TargetBucketName = ""; + this.TargetRegion = ""; + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/Replication/ReplicationLogData.java b/core/backend/src/com/pspace/backend/libs/Data/Replication/ReplicationLogData.java new file mode 100644 index 00000000..aecd2281 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/Replication/ReplicationLogData.java @@ -0,0 +1,55 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data.Replication; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class ReplicationLogData extends ReplicationEventData { + + public String Message; + + public ReplicationLogData() { + Init(); + } + + public ReplicationLogData(String Operation, String ObjectName, String VersionId, String SourceBucketName, + String TargetBucketName, String TargetRegion, String Message) { + super(Operation, ObjectName, VersionId, SourceBucketName, TargetBucketName, TargetRegion); + this.Message = Message; + } + + public ReplicationLogData(ReplicationEventData Event, String Message) { + super(Event); + this.Message = Message; + } + + @Override + public void Init() { + this.Operation = ""; + this.ObjectName = ""; + this.VersionId = ""; + this.SourceBucketName = ""; + this.TargetBucketName = ""; + this.TargetRegion = ""; + this.Message = ""; + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/S3/ReplicationData.java b/core/backend/src/com/pspace/backend/libs/Data/S3/ReplicationData.java new file mode 100644 index 00000000..0624005d --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/S3/ReplicationData.java @@ -0,0 +1,170 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data.S3; + +import java.util.ArrayList; +import java.util.Collection; + +import org.apache.commons.lang3.StringUtils; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.pspace.backend.libs.s3format.ReplicationConfiguration; +import com.pspace.backend.libs.s3format.S3Parameters; +import com.pspace.backend.libs.s3format.ReplicationConfiguration.Rule.Filter.Tag; + +public class ReplicationData { + + public boolean Status; + public boolean DeleteMarker; + public String TargetAccessControlTranslationOwner; + public String TargetAccount; + public String TargetBucket; + public String TargetRegion; + public int TargetMetricsEventThreshold; + public boolean TargetMetricsStatus; + public boolean TargetReplicationTimeStatus; + public int TargetReplicationTime; + public boolean ExistingObjectReplicationStatus; + public String Prefix; + public Collection Tags; + public String ID; + public int Priority; + public boolean ReplicaModificationsStatus; + + public boolean isFiltering; + + public ReplicationData() { + Init(); + } + + public void Init() { + Status = true; + DeleteMarker = false; + TargetAccessControlTranslationOwner = ""; + TargetAccount = ""; + TargetBucket = ""; + TargetRegion = ""; + TargetMetricsEventThreshold = 0; + TargetMetricsStatus = false; + TargetReplicationTimeStatus = false; + TargetReplicationTime = 0; + ExistingObjectReplicationStatus = false; + Prefix = ""; + Tags = new ArrayList(); + ID = ""; + Priority = 0; + ReplicaModificationsStatus = false; + isFiltering = false; + } + + public boolean Set(ReplicationConfiguration.Rule Conf) { + Init(); + if (Conf == null) return false; + if (Conf.destination == null) return false; + else { + // destination 설정이 비어있지 않을 경우 + ReplicationConfiguration.Rule.Destination destination = Conf.destination; + + // 복제 설정이 켜져 있다면 + if (S3Parameters.isEnabled(Conf.status)) Status = true; + else Status = false; + + TargetAccount = destination.account; + + // 버킷이름 파싱 + if (destination.bucket != null) + BucketNameParser(destination.bucket); + + // accessControlTranslation 설정이 있다면 + if (destination.accessControlTranslation != null) + TargetAccessControlTranslationOwner = destination.accessControlTranslation.owner; + + // metrics 설정이 있다면 + if (destination.metrics != null) { + ReplicationConfiguration.Rule.Destination.Metrics metrics = destination.metrics; + if (S3Parameters.isEnabled(metrics.status)) TargetMetricsStatus = true; + if (metrics.eventThreshold != null) TargetMetricsEventThreshold = metrics.eventThreshold.minutes; + } + + // replicationTime 설정이 있다면 + if (destination.replicationTime != null) { + ReplicationConfiguration.Rule.Destination.ReplicationTime replicationTime = Conf.destination.replicationTime; + if (S3Parameters.isEnabled(replicationTime.status)) TargetReplicationTimeStatus = true; + if (replicationTime.time != null) TargetReplicationTime = replicationTime.time.minutes; + } + } + + if (Conf.deleteMarkerReplication != null && S3Parameters.isEnabled(Conf.deleteMarkerReplication.Status)) DeleteMarker = true; + if (Conf.existingObjectReplication != null && S3Parameters.isEnabled(Conf.existingObjectReplication.status)) ExistingObjectReplicationStatus = true; + + /// Filter Priority + /// Filter.And : 1 + /// Filter : 2 + if (Conf.filter != null) { + isFiltering = true; + + // And 옵션이 있을 경우 가져오기 + if (Conf.filter.and != null) { + if (!StringUtils.isBlank(Conf.filter.and.prefix)) Prefix = Conf.filter.and.prefix; + if (Conf.filter.and.tag != null && Conf.filter.and.tag.size() > 0) Tags = Conf.filter.and.tag; + } + // And 설정이 없을 경우 적용 + else { + if (!StringUtils.isBlank(Conf.filter.prefix)) Prefix = Conf.filter.prefix; + if (Conf.filter.tag != null) Tags.add(Conf.filter.tag); + } + } + + ID = Conf.id; + Priority = Conf.priority; + + if (Conf.sourceSelectionCriteria != null) { + ReplicationConfiguration.Rule.SourceSelectionCriteria sourceSelectionCriteria = Conf.sourceSelectionCriteria; + if (sourceSelectionCriteria.replicaModifications != null) + if (S3Parameters.isEnabled(sourceSelectionCriteria.replicaModifications.status)) ReplicaModificationsStatus = true; + } + return true; + } + + private void BucketNameParser(String Bucket) { + + String Token[] = Bucket.split(":"); + + // 버킷이름을 파싱했으나 :이 없다면 + if (Token.length < 5) { + TargetBucket = Bucket; + return; + } + + // String ARN = Token[0]; + // String Partition = Token[1]; + // String Service = Token[2]; + String Region = Token[3]; + String BucketName = Token[5]; + + if (!StringUtils.isBlank(BucketName)) + TargetBucket = BucketName.trim(); + + //리전 정보가 존재할경우 리전 정보 저장 + if (!StringUtils.isBlank(Region)) + TargetRegion = Region.trim(); + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/S3/S3BucketData.java b/core/backend/src/com/pspace/backend/libs/Data/S3/S3BucketData.java new file mode 100644 index 00000000..cc708c16 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/S3/S3BucketData.java @@ -0,0 +1,170 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data.S3; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.lang3.StringUtils; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.pspace.backend.libs.Ksan.PortalManager; +import com.pspace.backend.libs.s3format.LoggingConfiguration; +import com.pspace.backend.libs.s3format.NotificationConfiguration; +import com.pspace.backend.libs.s3format.ReplicationConfiguration; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * + * @author KJW + */ +public class S3BucketData { + static final Logger logger = LoggerFactory.getLogger(S3BucketData.class); + + public String BucketName; + public LoggingConfiguration Loggings; + public NotificationConfiguration Notification; + public List Replications; + + public Boolean isLogging; + public Boolean isNotification; + public Boolean isReplication; + + public S3BucketData(String BucketName) { + this.BucketName = BucketName; + this.isLogging = false; + this.isNotification = false; + this.isReplication = false; + } + + public S3BucketData(com.pspace.ifs.ksan.objmanager.Bucket bucket) { + this.BucketName = bucket.getName(); + setLoggingConfiguration(bucket.getLogging()); + setReplicationConfiguration(bucket.getReplication()); + this.isNotification = false; + // setNotificationConfiguration(bucket.getNotification()); + } + + /** + * 로깅설정 텍스트를 언마샬링하여 필요한 설정을 추출한다. + * @param strLogging 로깅 설정 텍스트 + * @return 성공 / 실패 여부 + */ + public boolean setLoggingConfiguration(String strLogging) { + // 초기화 + isLogging = false; + Loggings = null; + + if (StringUtils.isBlank(strLogging)) + return false; + + try { + // 로깅 설정 언마샬링 + Loggings = new XmlMapper().readValue(strLogging, LoggingConfiguration.class); + if (Loggings == null) + return false; + if (Loggings.loggingEnabled == null) + return false; + + isLogging = true; + return true; + } catch (Exception e) { + logger.error("Bucket info read failed : {}", BucketName, e); + return false; + } + } + + /** + * 알림설정 텍스트를 언마샬링하여 필요한 설정을 추출한다. + * @param strNotification 알림 설정 텍스트 + * @return 성공 / 실패 여부 + */ + public boolean setNotificationConfiguration(String strNotification) { + // 초기화 + isNotification = false; + Notification = null; + + if (StringUtils.isBlank(strNotification)) + return false; + + try { + // 알림 설정 언마샬링 + Notification = new XmlMapper().readValue(strNotification, NotificationConfiguration.class); + if (Notification == null) + return false; + if (Notification.topics == null) + return false; + + isNotification = true; + return true; + } catch (Exception e) { + logger.error("Bucket info read failed : {}", BucketName, e); + return false; + } + } + + /** + * 복제설정 텍스트를 언마샬링하여 필요한 설정을 추출한다. + * @param strReplication 복제 설정 텍스트 + * @return 성공 / 실패 여부 + */ + public boolean setReplicationConfiguration(String strReplication) { + // 초기화 + isReplication = false; + if (Replications == null) + Replications = new ArrayList(); + else + Replications.clear(); + + if (StringUtils.isBlank(strReplication)) + return false; + + try { + // 복제 설정 언마샬링 + ReplicationConfiguration Replication = new XmlMapper().readValue(strReplication, + ReplicationConfiguration.class); + + if (Replication == null || Replication.rules == null || Replication.rules.size() == 0) + return false; + + for (var Role : Replication.rules) { + var Data = new ReplicationData(); + + if (Data != null && Data.Set(Role)) { + logger.debug("{}", Data.toString()); + Replications.add(Data); + } else + logger.error("Role is invalid : {}", BucketName); + } + isReplication = true; + return true; + } catch (Exception e) { + logger.error("Bucket info read failed : {}", BucketName, e); + return false; + } + } + + public void removeLoggingConfiguration() { + isLogging = false; + Loggings = null; + } + + public void removeNotificationConfiguration() { + isNotification = false; + Notification = null; + } + + public void removeReplicationConfiguration() { + isReplication = false; + Replications = null; + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/S3/S3LogData.java b/core/backend/src/com/pspace/backend/libs/Data/S3/S3LogData.java new file mode 100644 index 00000000..8b9f124a --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/S3/S3LogData.java @@ -0,0 +1,132 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data.S3; + +import java.util.ArrayList; +import java.util.List; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class S3LogData { + public String UserName; + public String BucketName; + public String Date; + public String RemoteHost; + public String RequestUser; + public String RequestId; + public String Operation; + public String ObjectName; + public String RequestURI; + public int StatusCode; + public String ErrorCode; + public long ResponseLength; + public long ObjectLength; + public long TotalTime; + public long RequestLength; + public String Referer; + public String UserAgent; + public String VersionId; + public String HostId; + public String Sign; + public String SSLGroup; + public String SignType; + public String EndPoint; + public String TLSVersion; + + public S3LogData(){ + Init(); + } + public S3LogData(String UserName, String BucketName, String Date, String Remote_host, + String RequestUser, String RequestId, String Operation, String ObjectName, String RequestURI, + int StatusCode, String ErrorCode, long ResponseLength, long ObjectLength, long TotalTime, + long RequestLength, String Referer, String UserAgent, String VersionId, String HostId, String Sign, + String SSLGroup, String SignType, String EndPoint, String TLSVersion) { + this.UserName = UserName; + this.BucketName = BucketName; + this.Date = Date; + this.RemoteHost = Remote_host; + this.RequestUser = RequestUser; + this.RequestId = RequestId; + this.Operation = Operation; + this.ObjectName = ObjectName; + this.RequestURI = RequestURI; + this.StatusCode = StatusCode; + this.ErrorCode = ErrorCode; + this.ResponseLength = ResponseLength; + this.ObjectLength = ObjectLength; + this.TotalTime = TotalTime; + this.RequestLength = RequestLength; + this.Referer = Referer; + this.UserAgent = UserAgent; + this.VersionId = VersionId; + this.HostId = HostId; + this.Sign = Sign; + this.SSLGroup = SSLGroup; + this.SignType = SignType; + this.EndPoint = EndPoint; + this.TLSVersion = TLSVersion; + } + + public void Init() { + UserName = ""; + BucketName = ""; + Date = ""; + RemoteHost = ""; + RequestUser = ""; + RequestId = ""; + Operation = ""; + ObjectName = ""; + RequestURI = ""; + StatusCode = 0; + ErrorCode = ""; + ResponseLength = 0; + ObjectLength = 0; + TotalTime = 0; + RequestLength = 0; + Referer = ""; + UserAgent = ""; + VersionId = ""; + HostId = ""; + Sign = ""; + SSLGroup = ""; + SignType = ""; + EndPoint = ""; + TLSVersion = ""; + } + + public boolean isOperationEmpty() { + return Operation.isEmpty() || Operation.equals("-"); + } + public boolean isBucketNameEmpty() { + return BucketName.isEmpty() || BucketName.equals("-"); + } + public boolean isError() { + return !ErrorCode.equals("-") && !ErrorCode.isBlank(); + } + + public String Print() { + return String.format("%s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s\n", + UserName, BucketName, Date, RemoteHost, RequestUser, RequestId, Operation, ObjectName, + RequestURI, StatusCode, ErrorCode, ResponseLength, ObjectLength, TotalTime, RequestLength, Referer, + UserAgent, VersionId, HostId, Sign, SSLGroup, SignType, EndPoint, TLSVersion); + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Data/S3/S3ObjectData.java b/core/backend/src/com/pspace/backend/libs/Data/S3/S3ObjectData.java new file mode 100644 index 00000000..2e3a5b65 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Data/S3/S3ObjectData.java @@ -0,0 +1,78 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Data.S3; + +import java.util.Collection; + +import org.apache.commons.lang3.StringUtils; + +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.pspace.backend.libs.s3format.Tagging; +import com.pspace.backend.libs.s3format.Tagging.TagSet.Tag; +import com.pspace.ifs.ksan.objmanager.Metadata; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class S3ObjectData { + final Logger logger = LoggerFactory.getLogger(S3ObjectData.class); + + public String ObjectName; + public String versionId; + public Collection tags; + + public Boolean isTagSet; + + public S3ObjectData(String ObjectName, String VersionId) { + this.ObjectName = ObjectName; + this.versionId = VersionId; + setTags(null); + } + + public S3ObjectData(String ObjectName, String VersionId, String StrTags) { + this.ObjectName = ObjectName; + this.versionId = VersionId; + setTags(StrTags); + } + + public S3ObjectData(Metadata Data) { + this.ObjectName = Data.getPath(); + this.versionId = Data.getVersionId(); + setTags(Data.getTag()); + } + + public Boolean setTags(String StrTagging) { + if (StringUtils.isBlank(StrTagging)) { + isTagSet = false; + return false; + } + try { + // Tag 언마샬링 + var MyTags = new XmlMapper().readValue(StrTagging, Tagging.class); + if (MyTags == null) + return false; + if (MyTags.tagset == null) + return false; + if (MyTags.tagset.tags == null) + return false; + if (MyTags.tagset.tags.size() == 0) + return false; + + for (var MyTag : MyTags.tagset.tags) + tags.add(MyTag); + isTagSet = true; + return true; + } catch (Exception e) { + logger.error("Object info read failed : {}", StrTagging, e); + return false; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Heartbeat/Heartbeat.java b/core/backend/src/com/pspace/backend/libs/Heartbeat/Heartbeat.java new file mode 100644 index 00000000..965c3e83 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Heartbeat/Heartbeat.java @@ -0,0 +1,48 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Heartbeat; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Data.Constants; +import com.pspace.ifs.ksan.libs.mq.MQSender; + +public class Heartbeat { + static final Logger logger = LoggerFactory.getLogger(Heartbeat.class); + + private static final String ONLINE = "Online"; + + private final String ServiceId; + private final MQSender Sender; + + public boolean Stop = false; + + public Heartbeat(String ServiceId, String Host, int Port, String User, String Password) + throws Exception { + this.ServiceId = ServiceId; + Sender = new MQSender(Host, Port, User, Password, Constants.MQ_KSAN_SYSTEM_EXCHANGE, "direct", ""); + } + + public void Start(int Delay) { + var Request = new RequestServiceState(ServiceId, ONLINE); + + while (!Stop) { + try { + Thread.sleep(Delay); + Sender.send(Request.toString(), Constants.MQ_HEARTBEAT_BINDING_KEY); + } catch (Exception e) { + logger.error("", e); + + } + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Heartbeat/RequestServiceState.java b/core/backend/src/com/pspace/backend/libs/Heartbeat/RequestServiceState.java new file mode 100644 index 00000000..84b4b210 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Heartbeat/RequestServiceState.java @@ -0,0 +1,35 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Heartbeat; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class RequestServiceState { + + public String Id; + public String State; + + public RequestServiceState(String Id, String State) { + this.Id = Id; + this.State = State; + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Ksan/AgentConfig.java b/core/backend/src/com/pspace/backend/libs/Ksan/AgentConfig.java new file mode 100644 index 00000000..4f78d542 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Ksan/AgentConfig.java @@ -0,0 +1,109 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Ksan.Data; + +import java.io.File; +import java.io.FileReader; + +import org.apache.commons.lang3.math.NumberUtils; +import org.ini4j.Ini; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.pspace.backend.libs.Data.Constants; + +public class AgentConfig { + /////////////////////////// MGS ///////////////////////////////////// + private final String STR_MGS = "mgs"; + private final String STR_PORTAL_IP = "PortalHost"; + private final String STR_PORTAL_PORT = "PortalPort"; + private final String STR_PORTAL_API_KEY = "PortalApiKey"; + private final String STR_MQ_IP = "MQHost"; + private final String STR_MQ_PORT = "MQPort"; + private final String STR_MQ_USER = "MQUser"; + private final String STR_MQ_PASSWORD = "MQPassword"; + /////////////////////////// MONITOR ///////////////////////////////////// + private final String STR_MONITOR = "monitor"; + private final String STR_SERVICE_MONITOR_INTERVAL = "ServiceMonitorInterval"; + + /*********************************************************************************************************/ + static final Logger logger = LoggerFactory.getLogger(AgentConfig.class); + private final Ini ini = new Ini(); + /*********************************************************************************************************/ + + public String PortalHost; + public int PortalPort; + public String MQHost; + public int MQPort; + public String MQUser; + public String MQPassword; + public String APIKey; + + public int ServiceMonitorInterval; + + public static AgentConfig getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static final AgentConfig INSTANCE = new AgentConfig(); + } + + public boolean getConfig() { + + File file = new File(Constants.AGENT_CONF_PATH); + try { + ini.load(new FileReader(file)); + + PortalHost = ReadKeyToString(STR_MGS, STR_PORTAL_IP); + PortalPort = ReadKeyToInt(STR_MGS, STR_PORTAL_PORT); + APIKey = ReadKeyToString(STR_MGS, STR_PORTAL_API_KEY); + + MQHost = ReadKeyToString(STR_MGS, STR_MQ_IP); + MQPort = ReadKeyToInt(STR_MGS, STR_MQ_PORT); + MQUser = ReadKeyToString(STR_MGS, STR_MQ_USER); + MQPassword = ReadKeyToString(STR_MGS, STR_MQ_PASSWORD); + + ServiceMonitorInterval = ReadKeyToInt(STR_MONITOR, STR_SERVICE_MONITOR_INTERVAL); + + } catch (Exception e) { + logger.error("", e); + return false; + } + return true; + } + + ////////////////////////////////////////////////////////////////////////// + private String ReadKeyToString(String Section, String Key) { + String Value = ini.get(Section, Key); + return (Value == null) ? "" : Value; + } + + private int ReadKeyToInt(String Section, String Key) { + return NumberUtils.toInt(ini.get(Section, Key)); + } + // private boolean ReadKeyToBoolean(String Section, String Key) + // { + // return Boolean.parseBoolean(ini.get(Section, Key)); + // } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Ksan/Data/QueryResults.java b/core/backend/src/com/pspace/backend/libs/Ksan/Data/QueryResults.java new file mode 100644 index 00000000..28b81c21 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Ksan/Data/QueryResults.java @@ -0,0 +1,62 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Ksan.Data; + +import java.util.List; + +import com.fasterxml.jackson.annotation.JsonProperty; + +public class QueryResults { + + @JsonProperty("HavePreviousPageSection") + public boolean HavePreviousPageSection; + + @JsonProperty("HaveNextPage") + public boolean HaveNextPage; + + @JsonProperty("HavePreviousPage") + public boolean HavePreviousPage; + + @JsonProperty("PageNos") + public List PageNos; + + @JsonProperty("EndPageNo") + public int EndPageNo; + + @JsonProperty("StartPageNo") + public int StartPageNo; + + @JsonProperty("PagePerSection") + public int PagePerSection; + + @JsonProperty("HaveNextPageSection") + public boolean HaveNextPageSection; + + @JsonProperty("CountPerPage") + public int CountPerPage; + + @JsonProperty("PageNo") + public int PageNo; + + @JsonProperty("Skips") + public int Skips; + + @JsonProperty("TotalCount") + public int TotalCount; + + @JsonProperty("TotalPage") + public int TotalPage; + + @JsonProperty("Items") + public List Items; + + +} diff --git a/core/backend/src/com/pspace/backend/libs/Ksan/Data/Response.java b/core/backend/src/com/pspace/backend/libs/Ksan/Data/Response.java new file mode 100644 index 00000000..16840998 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Ksan/Data/Response.java @@ -0,0 +1,30 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Ksan.Data; + +import com.fasterxml.jackson.annotation.JsonProperty; + +public class Response { + @JsonProperty("IsNeedLogin") + public boolean IsNeedLogin; + + @JsonProperty("AccessDenied") + public boolean AccessDenied; + + @JsonProperty("Result") + public String Result; + + @JsonProperty("Code") + public String Code; + + @JsonProperty("Message") + public String Message; +} diff --git a/portal/Portal/src/app/errors/401/error-401-page.component.ts b/core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseConfig.java similarity index 62% rename from portal/Portal/src/app/errors/401/error-401-page.component.ts rename to core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseConfig.java index dde9f4fa..729180b9 100644 --- a/portal/Portal/src/app/errors/401/error-401-page.component.ts +++ b/core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseConfig.java @@ -1,26 +1,27 @@ /* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ -import {Component, OnInit} from '@angular/core'; +package com.pspace.backend.libs.Ksan.Data; -@Component({ - selector: 'app-error-401-page', - templateUrl: './error-401-page.component.html', - styleUrls: ['./error-401-page.component.scss'] -}) -export class Error401PageComponent implements OnInit { +import com.fasterxml.jackson.annotation.JsonProperty; - constructor() { - } +public class ResponseConfig { + @JsonProperty("Type") + public String Type; - ngOnInit(): void { - } + @JsonProperty("Version") + public int Version; + @JsonProperty("Config") + public String Config; + + @JsonProperty("RegDate") + public String RegDate; } diff --git a/portal/Portal/src/app/shared/extensions/number.extension.ts b/core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseData.java similarity index 71% rename from portal/Portal/src/app/shared/extensions/number.extension.ts rename to core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseData.java index 1f335dca..2ca4f81d 100644 --- a/portal/Portal/src/app/shared/extensions/number.extension.ts +++ b/core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseData.java @@ -1,18 +1,18 @@ /* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ -/** - * 전역 확장 메서드 - */ -interface NumberConstructor { - MAX_VALUE_FOR_API: number; -} +package com.pspace.backend.libs.Ksan.Data; + +import com.fasterxml.jackson.annotation.JsonProperty; -Number.MAX_VALUE_FOR_API = 2147483647; +public class ResponseData extends Response{ + @JsonProperty("Data") + public T Data; +} diff --git a/portal/Portal/src/app/shared/classes/grid/grid-data-result-ex.ts b/core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseList.java similarity index 70% rename from portal/Portal/src/app/shared/classes/grid/grid-data-result-ex.ts rename to core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseList.java index 7798a50d..8f61fa65 100644 --- a/portal/Portal/src/app/shared/classes/grid/grid-data-result-ex.ts +++ b/core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseList.java @@ -1,16 +1,18 @@ /* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ -import {GridDataResult} from '@progress/kendo-angular-grid'; -import {Grid} from './grid.model'; +package com.pspace.backend.libs.Ksan.Data; -export interface GridDataResultEx extends GridDataResult { - grid: Grid; +import com.fasterxml.jackson.annotation.JsonProperty; + +public class ResponseList extends Response { + @JsonProperty("Data") + public QueryResults Data; } diff --git a/core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseRegion.java b/core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseRegion.java new file mode 100644 index 00000000..13f4f06a --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Ksan/Data/ResponseRegion.java @@ -0,0 +1,37 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Ksan.Data; + +import com.fasterxml.jackson.annotation.JsonProperty; + +public class ResponseRegion { + @JsonProperty("Name") + public String Name; + + @JsonProperty("Address") + public String Address; + + @JsonProperty("Port") + public int Port; + + @JsonProperty("SSLPort") + public int SSLPort; + + @JsonProperty("AccessKey") + public String AccessKey; + + @JsonProperty("SecretKey") + public String SecretKey; + + public String getURL() { + return String.format("http://%s:%s", Address, Port); + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Ksan/Data/S3RegionData.java b/core/backend/src/com/pspace/backend/libs/Ksan/Data/S3RegionData.java new file mode 100644 index 00000000..c29139ab --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Ksan/Data/S3RegionData.java @@ -0,0 +1,67 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Ksan.Data; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class S3RegionData { + public String Name; + public String Address; + public int Port; + public int SSLPort; + public String AccessKey; + public String SecretKey; + + public S3RegionData(String Name, String Address, int Port, int SSLPort, String AccessKey, String SecretKey) { + this.Name = Name; + this.Address = Address; + this.Port = Port; + this.SSLPort = SSLPort; + this.AccessKey = AccessKey; + this.SecretKey = SecretKey; + } + public S3RegionData(ResponseRegion Data) { + this.Name = Data.Name; + this.Address = Data.Address; + this.Port = Data.Port; + this.SSLPort = Data.SSLPort; + this.AccessKey = Data.AccessKey; + this.SecretKey = Data.SecretKey; + } + + public void init() { + Name = ""; + Address = ""; + Port = 0; + SSLPort = 0; + AccessKey = ""; + SecretKey = ""; + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } + + public String getHttpURL() { + return String.format("http://%s:%d", Address, Port); + } + + public String getHttpsURL() { + return String.format("https://%s:%d", Address, SSLPort); + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Ksan/ObjManagerHelper.java b/core/backend/src/com/pspace/backend/libs/Ksan/ObjManagerHelper.java new file mode 100644 index 00000000..bd60b61e --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Ksan/ObjManagerHelper.java @@ -0,0 +1,55 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Ksan; + +import java.sql.SQLException; +import java.util.List; + +import com.pspace.backend.libs.Data.S3.S3BucketData; +import com.pspace.backend.libs.Data.S3.S3ObjectData; +import com.pspace.ifs.ksan.objmanager.Bucket; +import com.pspace.ifs.ksan.objmanager.ObjManagerConfig; +import com.pspace.ifs.ksan.objmanager.ObjManagerUtil; +import com.pspace.ifs.ksan.objmanager.ObjManagerException.ResourceNotFoundException; + +public class ObjManagerHelper { + private static ObjManagerUtil ObjManager; + + public static ObjManagerHelper getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static final ObjManagerHelper INSTANCE = new ObjManagerHelper(); + } + + public void init(ObjManagerConfig config) throws Exception + { + ObjManager = new ObjManagerUtil(config); + } + + public List getBucketList() + { + return ObjManager.getBucketList(); + } + + public S3BucketData getBucket(String bucketName) throws ResourceNotFoundException, SQLException + { + return new S3BucketData(ObjManager.getBucket(bucketName)); + } + + public S3ObjectData getObject(String bucketName, String objectName) throws ResourceNotFoundException{ + return new S3ObjectData(ObjManager.getObject(bucketName, objectName)); + } + public S3ObjectData getObject(String bucketName, String objectName, String versionId) throws ResourceNotFoundException{ + return new S3ObjectData(ObjManager.getObject(bucketName, objectName, versionId)); + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Ksan/PortalManager.java b/core/backend/src/com/pspace/backend/libs/Ksan/PortalManager.java new file mode 100644 index 00000000..aad7e1fe --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Ksan/PortalManager.java @@ -0,0 +1,226 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.Ksan; + +import java.security.KeyManagementException; +import java.security.KeyStoreException; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.lang3.StringUtils; +import org.apache.http.client.ResponseHandler; +import org.apache.http.client.methods.HttpGet; +import org.apache.http.conn.ssl.NoopHostnameVerifier; +import org.apache.http.conn.ssl.TrustAllStrategy; +import org.apache.http.impl.client.BasicResponseHandler; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.ssl.SSLContextBuilder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.pspace.backend.libs.Config.LogManagerConfig; +import com.pspace.backend.libs.Config.ReplicationConfig; +import com.pspace.backend.libs.Ksan.Data.ResponseConfig; +import com.pspace.backend.libs.Ksan.Data.ResponseData; +import com.pspace.backend.libs.Ksan.Data.ResponseList; +import com.pspace.backend.libs.Ksan.Data.ResponseRegion; +import com.pspace.backend.libs.Ksan.Data.S3RegionData; +import com.pspace.backend.libs.Ksan.Data.AgentConfig; + +public class PortalManager { + static final Logger logger = LoggerFactory.getLogger(PortalManager.class); + private AgentConfig Config; + + private List regions = null; + + public static PortalManager getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static final PortalManager INSTANCE = new PortalManager(); + } + + private PortalManager() { + Config = AgentConfig.getInstance(); + } + + /** + * Replication 설정 정보를 가져온다. + * + * @return Replication 설정 정보 + */ + public ReplicationConfig getReplicationConfig() { + try (var Client = getClient();) { + + var get = getRequest(getReplicationConfigURL()); + var Response = Client.execute(get); + if (Response.getStatusLine().getStatusCode() == 200) { + ResponseHandler handler = new BasicResponseHandler(); + var Body = handler.handleResponse(Response); + logger.debug("Body : {}", Body); + + var Mapper = new ObjectMapper(); + var Result = Mapper.readValue(Body, new TypeReference>() { + }); + + if (!Result.Code.isEmpty()) { + logger.error("{}, {}", Result.Code, Result.Message); + return null; + } + + return Mapper.readValue(Result.Data.Config, ReplicationConfig.class); + } + + } catch (Exception e) { + logger.error("", e); + } + return null; + } + + /** + * LogManager 설정 정보를 가져온다. + * + * @return LogManager 설정 정보 + */ + public LogManagerConfig getLogManagerConfig() { + try (var Client = getClient();) { + + var get = getRequest(getLogManagerConfigURL()); + var Response = Client.execute(get); + if (Response.getStatusLine().getStatusCode() == 200) { + ResponseHandler handler = new BasicResponseHandler(); + var Body = handler.handleResponse(Response); + logger.debug("Body : {}", Body); + + var Mapper = new ObjectMapper(); + var Result = Mapper.readValue(Body, new TypeReference>() { + }); + + if (!Result.Code.isEmpty()) { + logger.error("{}, {}", Result.Code, Result.Message); + return null; + } + + return Mapper.readValue(Result.Data.Config, LogManagerConfig.class); + } + + } catch (Exception e) { + logger.error("", e); + } + return null; + } + + public boolean RegionInit() { + try (var Client = getClient();) { + + var get = getRequest(getRegionsURL()); + var Response = Client.execute(get); + + if (Response.getStatusLine().getStatusCode() == 200) { + ResponseHandler handler = new BasicResponseHandler(); + var Body = handler.handleResponse(Response); + + var Mapper = new ObjectMapper(); + var Result = Mapper.readValue(Body, new TypeReference>() { + }); + + if (!Result.Code.isEmpty()) { + logger.error("region is empty. [{}, {}]", Result.Code, Result.Message); + return false; + } + + if (regions == null) { + regions = new ArrayList(); + for (var Region : Result.Data.Items) + regions.add(new S3RegionData(Region)); + } + } + } catch (Exception e) { + logger.error("", e); + return false; + } + return true; + } + + /** + * URL을 바탕으로 리전정보를 조회한다. + * + * @param Address 찾을 리전 주소 + * @return 리전 정보 + */ + public S3RegionData get2Address(String Address) { + if (regions == null || regions.size() == 0 || StringUtils.isBlank(Address)) + return null; + return regions.stream().filter(f -> Address.equals(f.Address)).findAny().orElse(null); + } + + /** + * 리전명을 바탕으로 리전정보를 조회한다. + * + * @param RegionName 찾을 리전 명 + * @return 리전 정보 + */ + public S3RegionData getRegion(String RegionName) { + if (regions == null || regions.size() == 0 || StringUtils.isBlank(RegionName)) + return null; + return regions.stream().filter(f -> RegionName.equals(f.Name)).findAny().orElse(null); + } + + private String getURL() { + return String.format("https://%s:%s", Config.PortalHost, Config.PortalPort); + } + + private String getBackendConfigURL() { + return String.format("%s/api/v1/Config/KsanS3Backend", getURL()); + } + + private String getLifecycleConfigURL() { + return String.format("%s/api/v1/Config/KsanLifecycleManager", getURL()); + } + + private String getReplicationConfigURL() { + return String.format("%s/api/v1/Config/KsanReplicationManager", getURL()); + } + + private String getLogManagerConfigURL() { + return String.format("%s/api/v1/Config/KsanLogManager", getURL()); + } + + private String getMeteringConfigURL() { + return String.format("%s/api/v1/Config/KsanS3Backend", getURL()); + } + + private String getRegionsURL() { + return String.format("%s/api/v1/Regions", getURL()); + } + + private String getRegionURL(String RegionName) { + return String.format("%s/api/v1/Regions/%s", getURL(), RegionName); + } + + private HttpGet getRequest(String URL) { + var Request = new HttpGet(URL); + Request.addHeader("Authorization", Config.APIKey); + return Request; + } + + private CloseableHttpClient getClient() throws KeyManagementException, NoSuchAlgorithmException, KeyStoreException { + return HttpClients.custom() + .setSSLContext(new SSLContextBuilder().loadTrustMaterial(null, TrustAllStrategy.INSTANCE).build()) + .setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE) + .build(); + } +} diff --git a/core/backend/src/com/pspace/backend/libs/MQ/MQManager.java b/core/backend/src/com/pspace/backend/libs/MQ/MQManager.java new file mode 100644 index 00000000..e883218f --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/MQ/MQManager.java @@ -0,0 +1,51 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.MQ; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Ksan.Data.AgentConfig; +import com.pspace.ifs.ksan.libs.mq.MQSender; + +public class MQManager { + static final Logger logger = LoggerFactory.getLogger(MQManager.class); + private static AgentConfig Config; + private static MQSender Sender; + + public static MQManager getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static final MQManager INSTANCE = new MQManager(); + } + + public MQManager() { + Config = AgentConfig.getInstance(); + } + + public void init() throws Exception { + if (Sender == null) + Sender = new MQSender(Config.MQHost, Config.MQPort, Config.MQUser, Config.MQPassword, + Constants.MQ_KSAN_LOG_EXCHANGE, Constants.MQ_EXCHANGE_OPTION_DIRECT, ""); + } + public void init(String exchangeOption, String routingKey) throws Exception { + if (Sender == null) + Sender = new MQSender(Config.MQHost, Config.MQPort, Config.MQUser, Config.MQPassword, + Constants.MQ_KSAN_LOG_EXCHANGE, exchangeOption, routingKey); + } + + public void Send(String Message, String routingKey) throws Exception { + Sender.send(Message, routingKey); + } +} diff --git a/core/backend/src/com/pspace/backend/libs/Utility.java b/core/backend/src/com/pspace/backend/libs/Utility.java new file mode 100644 index 00000000..b68ad760 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/Utility.java @@ -0,0 +1,82 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs; + +import java.io.BufferedReader; +import java.io.FileReader; +import java.io.ByteArrayInputStream; +import java.io.FileWriter; +import java.io.InputStream; +import java.io.File; +import java.lang.management.ManagementFactory; +import java.text.SimpleDateFormat; +import java.util.Date; + +import org.apache.http.client.HttpClient; +import org.apache.http.client.config.RequestConfig; +import org.apache.http.client.methods.HttpGet; +import org.apache.http.impl.client.HttpClientBuilder; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class Utility { + static final Logger logger = LoggerFactory.getLogger(Utility.class); + private static final int TimeOut = 3 * 1000; + private static final SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy.MM.dd HH:mm:ss"); + + public static String ReadServiceId(String FilePath) { + try { + BufferedReader Reader = new BufferedReader(new FileReader(FilePath)); + var ServiceId = Reader.readLine(); + Reader.close(); + return ServiceId; + } catch (Exception e) { + return null; + } + } + + public static String GetNowTime() + { + var now = new Date(); + return simpleDateFormat.format(now); + } + + public static void Delay(int milliseconds) { + try { + Thread.sleep(milliseconds); + } catch (InterruptedException e) { + } + } + + public static boolean S3AliveCheck(String URL) { + try { + HttpClient Client = HttpClientBuilder.create().build(); + HttpGet getRequest = new HttpGet(URL); + RequestConfig requestConfig = RequestConfig.custom() + .setSocketTimeout(TimeOut) + .setConnectTimeout(TimeOut) + .setConnectionRequestTimeout(TimeOut) + .build(); + getRequest.setConfig(requestConfig); + Client.execute(getRequest); + getRequest.releaseConnection(); + return true; + } catch (Exception e) { + logger.error("URL : {}", URL, e); + } + return false; + } + + public static InputStream CreateBody(String Body) { + return new ByteArrayInputStream(Body.getBytes()); + } +} \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/libs/docker/DockerFile b/core/backend/src/com/pspace/backend/libs/docker/DockerFile new file mode 100644 index 00000000..51808490 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/docker/DockerFile @@ -0,0 +1,4 @@ +FROM maven:3.8.6-jdk-11 as backendlibsbuild +WORKDIR /src +COPY ./ /src/ +RUN --mount=type=cache,target=/root/.m2 mvn -f /src/pom.xml install \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/libs/pom.xml b/core/backend/src/com/pspace/backend/libs/pom.xml new file mode 100644 index 00000000..2efe3e13 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/pom.xml @@ -0,0 +1,89 @@ + + 4.0.0 + com.pspace.backend.libs + ksan-backend-libs + 1.1.1 + jar + ksan-backend-libs + + + 1.7.30 + UTF-8 + 11 + 2.12.5 + + + + + org.apache.commons + commons-lang3 + 3.12.0 + + + + com.pspace.ifs.ksan.libs + ksan-libs + 0.7.1 + + + com.pspace.ifs.ksan.objmanager + ksan-objmanager + 0.7.1 + + + + com.fasterxml.jackson.dataformat + jackson-dataformat-xml + 2.10.3 + + + com.fasterxml.jackson.core + jackson-databind + 2.11.1 + + + + org.ini4j + ini4j + 0.5.4 + + + + ch.qos.logback + logback-classic + 1.2.3 + + + org.slf4j + slf4j-api + ${slf4j.version} + + + org.slf4j + slf4j-simple + ${slf4j.version} + + + com.rabbitmq + amqp-client + 5.9.0 + + + + + . + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${jdk.version} + ${jdk.version} + UTF-8 + + + + + diff --git a/core/backend/src/com/pspace/backend/libs/s3format/LoggingConfiguration.java b/core/backend/src/com/pspace/backend/libs/s3format/LoggingConfiguration.java new file mode 100644 index 00000000..4e8072fc --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/s3format/LoggingConfiguration.java @@ -0,0 +1,61 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.s3format; + +import java.util.Collection; + +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper; +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; + +/** Represent an Amazon LoggingConfiguration. */ +// CHECKSTYLE:OFF +public final class LoggingConfiguration { + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "LoggingEnabled") + public LoggingEnabled loggingEnabled; + + public static final class LoggingEnabled { + @JacksonXmlProperty(localName = "TargetBucket") + public String targetBucket; + + @JacksonXmlProperty(localName = "TargetPrefix") + public String targetPrefix; + + @JacksonXmlProperty(localName = "TargetGrants") + public TargetGrants targetGrants; + + public static final class TargetGrants { + @JacksonXmlProperty(localName = "Grant") + public Collection grants; + + public static final class Grant { + @JacksonXmlProperty(localName = "Grantee") + public Grantee grantee; + @JacksonXmlProperty(localName = "Permission") + public String permission; + + public static final class Grantee { + @JacksonXmlProperty(namespace = "http://www.w3.org/2001/XMLSchema-instance", localName = "type", isAttribute = true) + public String type; + @JacksonXmlProperty(localName = "ID") + public String id; + @JacksonXmlProperty(localName = "DisplayName") + public String displayName; + @JacksonXmlProperty(localName = "EmailAddress") + public String emailAddress; + @JacksonXmlProperty(localName = "URI") + public String uri; + } + } + } + } +} +// CHECKSTYLE:ON diff --git a/core/backend/src/com/pspace/backend/libs/s3format/NotificationConfiguration.java b/core/backend/src/com/pspace/backend/libs/s3format/NotificationConfiguration.java new file mode 100644 index 00000000..aa24b516 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/s3format/NotificationConfiguration.java @@ -0,0 +1,132 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.s3format; + +import java.util.Collection; + +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper; +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; + +/** Represent an Amazon NotificationConfiguration. */ +// CHECKSTYLE:OFF +public final class NotificationConfiguration { + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "TopicConfiguration") + public Collection topics; + + public static final class TopicConfiguration { + @JacksonXmlProperty(localName = "Event") + public String event; + + @JacksonXmlProperty(localName = "Id") + public String Id; + + @JacksonXmlProperty(localName = "Topic") + public String topic; + + @JacksonXmlProperty(localName = "Filter") + public Filter filter; + + public static final class Filter { + @JacksonXmlProperty(localName = "S3Key") + public S3Key s3key; + + public static final class S3Key { + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "FilterRule") + public Collection filterrule; + + public static final class FilterRule { + @JacksonXmlProperty(localName = "Name") + public String name; + + @JacksonXmlProperty(localName = "Value") + public String value; + } + } + } + } + + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "QueueConfiguration") + public Collection queue; + + public static final class QueueConfiguration { + @JacksonXmlProperty(localName = "Event") + public String event; + + @JacksonXmlProperty(localName = "Id") + public String Id; + + @JacksonXmlProperty(localName = "Queue") + public String queue; + + @JacksonXmlProperty(localName = "Filter") + public Filter filter; + + public static final class Filter { + @JacksonXmlProperty(localName = "S3Key") + public S3Key s3key; + + public static final class S3Key { + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "FilterRule") + public Collection filterrule; + + public static final class FilterRule { + @JacksonXmlProperty(localName = "Name") + public String name; + + @JacksonXmlProperty(localName = "Value") + public String value; + } + } + } + } + + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "CloudFunctionConfiguration") + public Collection cloud; + + public static final class CloudFunctionConfiguration { + @JacksonXmlProperty(localName = "Event") + public String event; + + @JacksonXmlProperty(localName = "Id") + public String Id; + + @JacksonXmlProperty(localName = "CloudFunction") + public String cloudfunction; + + @JacksonXmlProperty(localName = "Filter") + public Filter filter; + + public static final class Filter { + @JacksonXmlProperty(localName = "S3Key") + public S3Key s3key; + + public static final class S3Key { + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "FilterRule") + public Collection filterrule; + + public static final class FilterRule { + @JacksonXmlProperty(localName = "Name") + public String name; + + @JacksonXmlProperty(localName = "Value") + public String value; + } + } + } + } +} +// CHECKSTYLE:ON diff --git a/core/backend/src/com/pspace/backend/libs/s3format/ReplicationConfiguration.java b/core/backend/src/com/pspace/backend/libs/s3format/ReplicationConfiguration.java new file mode 100644 index 00000000..acf42a9c --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/s3format/ReplicationConfiguration.java @@ -0,0 +1,181 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.s3format; + +import java.util.Collection; + +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper; +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; + +/** Represent an Amazon ReplicationConfiguration. */ +// CHECKSTYLE:OFF +public final class ReplicationConfiguration { + @JacksonXmlProperty(localName = "Role") + public String role; + + @JacksonXmlProperty(localName = "Rule") + @JacksonXmlElementWrapper(useWrapping = false) + public Collection rules; + + public static final class Rule { + + @JacksonXmlProperty(localName = "DeleteMarkerReplication") + public DeleteMarkerReplication deleteMarkerReplication; + + public static final class DeleteMarkerReplication { + @JacksonXmlProperty(localName = "Status") + public String Status; + } + + @JacksonXmlProperty(localName = "Destination") + public Destination destination; + + public static final class Destination { + @JacksonXmlProperty(localName = "AccessControlTranslation") + public AccessControlTranslation accessControlTranslation; + + public static final class AccessControlTranslation { + @JacksonXmlProperty(localName = "Owner") + public String owner; + } + + @JacksonXmlProperty(localName = "Account") + public String account; + + @JacksonXmlProperty(localName = "Bucket") + public String bucket; + + @JacksonXmlProperty(localName = "EncryptionConfiguration") + public EncryptionConfiguration encryptionConfiguration; + + public static final class EncryptionConfiguration { + @JacksonXmlProperty(localName = "ReplicaKmsKeyID") + public String replicaKmsKeyID; + } + + @JacksonXmlProperty(localName = "Metrics") + public Metrics metrics; + + public static final class Metrics { + @JacksonXmlProperty(localName = "EventThreshold") + public EventThreshold eventThreshold; + + public static final class EventThreshold { + @JacksonXmlProperty(localName = "Minutes") + public int minutes; + } + + @JacksonXmlProperty(localName = "Status") + public String status; + } + + @JacksonXmlProperty(localName = "ReplicationTime") + public ReplicationTime replicationTime; + + public static final class ReplicationTime { + @JacksonXmlProperty(localName = "Time") + public Time time; + + public static final class Time { + @JacksonXmlProperty(localName = "Minutes") + public int minutes; + } + + @JacksonXmlProperty(localName = "Status") + public String status; + } + + @JacksonXmlProperty(localName = "StorageClass") + public String storageClass; + } + + @JacksonXmlProperty(localName = "ExistingObjectReplication") + public ExistingObjectReplication existingObjectReplication; + + public static final class ExistingObjectReplication { + @JacksonXmlProperty(localName = "Status") + public String status; + } + + @JacksonXmlProperty(localName = "Filter") + public Filter filter; + + public static final class Filter { + @JacksonXmlProperty(localName = "And") + public And and; + + public static final class And { + @JacksonXmlProperty(localName = "Prefix") + public String prefix; + + @JacksonXmlProperty(localName = "Tag") + @JacksonXmlElementWrapper(useWrapping = false) + public Collection tag; + + public static final class Tag { + @JacksonXmlProperty(localName = "Key") + public String key; + + @JacksonXmlProperty(localName = "Value") + public String value; + } + } + + @JacksonXmlProperty(localName = "Prefix") + public String prefix; + + @JacksonXmlProperty(localName = "Tag") + public Tag tag; + + public static final class Tag { + @JacksonXmlProperty(localName = "Key") + public String key; + + @JacksonXmlProperty(localName = "Value") + public String value; + } + } + + @JacksonXmlProperty(localName = "ID") + public String id; + + @JacksonXmlProperty(localName = "Prefix") + public String prefix; + + @JacksonXmlProperty(localName = "Priority") + public int priority; + + @JacksonXmlProperty(localName = "SourceSelectionCriteria") + public SourceSelectionCriteria sourceSelectionCriteria; + + public static final class SourceSelectionCriteria { + @JacksonXmlProperty(localName = "ReplicaModifications") + public ReplicaModifications replicaModifications; + + public static final class ReplicaModifications { + @JacksonXmlProperty(localName = "Status") + public String status; + } + + @JacksonXmlProperty(localName = "SseKmsEncryptedObjects") + public SseKmsEncryptedObjects sseKmsEncryptedObjects; + + public static final class SseKmsEncryptedObjects { + @JacksonXmlProperty(localName = "Status") + public String status; + } + } + + @JacksonXmlProperty(localName = "Status") + public String status; + } +} +// CHECKSTYLE:ON \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/libs/s3format/S3Parameters.java b/core/backend/src/com/pspace/backend/libs/s3format/S3Parameters.java new file mode 100644 index 00000000..8bba1b13 --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/s3format/S3Parameters.java @@ -0,0 +1,163 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.s3format; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +public final class S3Parameters { + public static final String ENABLED = "Enabled"; + public static final String DISABLED = "Disabled"; + public static final String PUT = "PUT"; + public static final String DELETE = "DELETE"; + + public static final String OP_DELETE_WEBSITE = "REST.DELETE.WEBSITE"; + public static final String OP_DELETE_POLICY = "REST.DELETE.POLICY"; + public static final String OP_DELETE_CORS = "REST.DELETE.CORS"; + public static final String OP_DELETE_LIFECYCLE = "REST.DELETE.LIFECYCLE"; + public static final String OP_DELETE_PUBLICACCESSBLOCK = "REST.DELETE.PUBLICACCESSBLOCK"; + public static final String OP_DELETE_BUCKET_TAGGING = "REST.DELETE.BUCKET.TAGGING"; + public static final String OP_DELETE_ENCRYPTION = "REST.DELETE.ENCRYPTION"; + public static final String OP_DELETE_LOGGING = "REST.DELETE.LOGGING"; + public static final String OP_DELETE_BUCKET = "REST.DELETE.BUCKET"; + public static final String OP_DELETE_OBJECT_UPLOAD = "REST.DELETE.OBJECT.UPLOAD"; + public static final String OP_DELETE_OBJECT_TAGGING = "REST.DELETE.OBJECT.TAGGING"; + public static final String OP_DELETE_OBJECT = "REST.DELETE.OBJECT"; + public static final String OP_DELETE_OBJECTLOCK = "REST.DELETE.OBJECTLOCK"; + public static final String OP_DELETE_NOTIFICATION = "REST.DELETE.NOTIFICATION"; + public static final String OP_DELETE_REPLICATION = "REST.DELETE.REPLICATION"; + + public static final String OP_GET_LISTBUCKET = "REST.GET.LISTBUCKET"; + public static final String OP_GET_WEBSITE = "REST.GET.WEBSITE"; + public static final String OP_GET_POLICY = "REST.GET.POLICY"; + public static final String OP_GET_CORS = "REST.GET.CORS"; + public static final String OP_GET_LIFECYCLE = "REST.GET.LIFECYCLE"; + public static final String OP_GET_PUBLICACCESSBLOCK = "REST.GET.PUBLICACCESSBLOCK"; + public static final String OP_GET_BUCKET_TAGGING = "REST.GET.BCUKET.TAGGING"; + public static final String OP_GET_LOGGING = "REST.GET.LOGGING"; + public static final String OP_GET_OBJECTLOCK = "REST.GET.OBJECTLOCK"; + public static final String OP_GET_NOTIFICATION = "REST.GET.NOTIFICATION"; + public static final String OP_GET_BUCKET_POLICY_STATUS = "REST.GET.BUCKET.POLICY.STATUS"; + public static final String OP_GET_REPLICATION = "REST.GET.REPLICATION"; + + public static final String OP_GET_ENCRYPTION = "REST.GET.ENCRYPTION"; + public static final String OP_GET_BUCKET_ACL = "REST.GET.BUCKET.ACL"; + public static final String OP_GET_LOCATION = "REST.GET.LOCATION"; + public static final String OP_GET_UPLOADS = "REST.GET.UPLOADS"; + public static final String OP_GET_VERSIONING = "REST.GET.VERSIONING"; + public static final String OP_GET_LISTOBJECTSV2 = "REST.GET.LISTOBJECTV2"; + public static final String OP_GET_LISTOBJECTS = "REST.GET.LISTOBJECT"; + public static final String OP_GET_LISTVERSIONS = "REST.GET.LISTVERSIONS"; + public static final String OP_GET_OBJECT_ACL = "REST.GET.OBJECT.ACL"; + public static final String OP_GET_OBJECT_RETENTION = "REST.GET.OBJECT.RETENTION"; + public static final String OP_GET_OBJECT_LEGAL_HOLD = "REST.GET.OBJECT.LEGAL.HOLD"; + public static final String OP_GET_OBJECT_LISTPARTS = "REST.GET.OBJECT.LISTPARTS"; + public static final String OP_GET_OBJECT_TAGGING = "REST.GET.OBJECT.TAGGING"; + public static final String OP_GET_OBJECT = "REST.GET.OBJECT"; + + public static final String OP_HEAD_BUCKET = "REST.HEAD.BUCKET"; + public static final String OP_HEAD_OBJECT = "REST.HEAD.OBJECT"; + + public static final String OP_POST_DELETEOBJECTS = "REST.POST.DELETEOBJECTS"; + public static final String OP_POST_UPLOAD = "REST.POST.UPLOAD"; + public static final String OP_POST_COMPLETE = "REST.POST.COMPLETEUPLOAD"; + public static final String OP_POST_OBJECT = "REST.POST.OBJECT"; + + public static final String OP_PUT_WEBSITE = "REST.PUT.WEBSITE"; + public static final String OP_PUT_POLICY = "REST.PUT.POLICY"; + public static final String OP_PUT_CORS = "REST.PUT.CORS"; + public static final String OP_PUT_LIFECYCLE = "REST.PUT.LIFECYCLE"; + public static final String OP_PUT_PUBLICACCESSBLOCK = "REST.PUT.PUBLICACCESSBLOCK"; + public static final String OP_PUT_BUCKET_TAGGING = "REST.PUT.BUCKET.TAGGING"; + public static final String OP_PUT_LOGGING = "REST.PUT.LOGGING"; + public static final String OP_PUT_ENCRYPTION = "REST.PUT.ENCRYPTION"; + public static final String OP_PUT_BUCKET_ACL = "REST.PUT.BUCKET.ACL"; + public static final String OP_PUT_VERSIONING = "REST.PUT.VERSIONING"; + public static final String OP_PUT_BCUKET = "REST.PUT.BUCKET"; + public static final String OP_PUT_OBJECTLOCK = "REST.PUT.OBJECTLOCK"; + public static final String OP_PUT_NOTIFICATION = "REST.PUT.NOTIFICATION"; + public static final String OP_PUT_REPLICATION = "REST.PUT.REPLICATION"; + + public static final String OP_PUT_OBJECT_PART_COPY = "REST.PUT.OBJECT.PART.COPY"; + public static final String OP_PUT_OBJECT_PART = "REST.PUT.OBJECT.PART"; + public static final String OP_PUT_OBJECT_COPY = "REST.PUT.OBJECT.COPY"; + public static final String OP_PUT_OBJECT_RETENTION = "REST.PUT.OBJECT.RETENTION"; + public static final String OP_PUT_OBJECT_LEGAL_HOLD = "REST.PUT.OBJECT.LEGAL.HOLD"; + public static final String OP_PUT_OBJECT_TAGGING = "REST.PUT.OBJECT.TAGGING"; + public static final String OP_PUT_OBJECT_ACL = "REST.PUT.OBJECT.ACL"; + public static final String OP_PUT_OBJECT = "REST.PUT.OBJECT"; + + public static boolean isEnabled(String Status) { + return ENABLED.equalsIgnoreCase(Status); + } + + public static boolean isDisabled(String Status) { + return DISABLED.equalsIgnoreCase(Status); + } + + // 해당 작업이 추가인지 확인 + public static boolean PutOperationCheck(String Operation) { + // 이름에 PUT이 포함되어 있지 않을 경우 false 반환 + if (Operation.indexOf(PUT) < 0) + return false; + return true; + } + + // 해당 작업이 삭제인지 확인 + public static boolean DeleteOperationCheck(String Operation) { + // 이름에 DELETE가 포함되어 있지 않을 경우 false 반환 + if (Operation.indexOf(DELETE) < 0) + return false; + return true; + } + + /************************************* + * Bucket Info Change Event + **************************************/ + // 버킷의 설정이 변경되는 작업 목록 + public static final List BUCKET_INFO_CHANGED = new ArrayList(Arrays.asList(new String[] { + OP_PUT_LOGGING, OP_PUT_NOTIFICATION, OP_PUT_REPLICATION, + OP_DELETE_BUCKET, OP_DELETE_LOGGING, OP_DELETE_NOTIFICATION, OP_DELETE_REPLICATION })); + + // 버킷의 설정이 변경되었는지 확인 + public static boolean BucketInfoChanged(String Operation) { + if (BUCKET_INFO_CHANGED.contains(Operation)) + return true; + return false; + } + + // 버킷의 설정이 추가되는 작업 목록 + public static final List BUCKET_INFO_UPDATE = new ArrayList(Arrays.asList(new String[] { + OP_PUT_LOGGING, OP_PUT_NOTIFICATION, OP_PUT_REPLICATION })); + + // 버킷의 설정이 추가되는지 확인 + public static boolean BucketInfoUpdateCheck(String Operation) { + if (BUCKET_INFO_UPDATE.contains(Operation)) + return true; + return false; + } + + /******************************************** + * Replication + ********************************************/ + // 복제해야할 작업 목록 + public static final List REPLICATE_OPERATION_LIST = new ArrayList<>(Arrays.asList(new String[] { + OP_PUT_OBJECT, OP_PUT_OBJECT_COPY, OP_POST_COMPLETE, OP_PUT_OBJECT_ACL, OP_PUT_OBJECT_RETENTION, OP_PUT_OBJECT_TAGGING, OP_DELETE_OBJECT, + OP_DELETE_OBJECT_TAGGING })); + + // 복제해야할 작업 확인 + public static boolean ReplicateOperationCheck(String Operation) { + if (REPLICATE_OPERATION_LIST.contains(Operation)) + return true; + return false; + } +} \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/libs/s3format/Tagging.java b/core/backend/src/com/pspace/backend/libs/s3format/Tagging.java new file mode 100644 index 00000000..24b10a5c --- /dev/null +++ b/core/backend/src/com/pspace/backend/libs/s3format/Tagging.java @@ -0,0 +1,63 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.libs.s3format; + +import java.util.Collection; + +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper; +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; + +/** Represent an Amazon Tagging for a bucket or object. */ +// CHECKSTYLE:OFF +public final class Tagging { + @JacksonXmlProperty(localName = "TagSet") + public TagSet tagset; + + @Override + public String toString() { + StringBuilder sb = new StringBuilder(); + sb.append(TagSet.class.getSimpleName()).append('{'); + sb.append("tagset=").append(tagset); + return sb.append('}').toString(); + } + + public static final class TagSet { + @JacksonXmlElementWrapper(useWrapping = false) + @JacksonXmlProperty(localName = "Tag") + public Collection tags; + + @Override + public String toString() { + StringBuilder sb = new StringBuilder(); + sb.append(Tag.class.getSimpleName()).append('{'); + sb.append("tags=").append(tags); + return sb.append('}').toString(); + } + + public static final class Tag { + @Override + public String toString() { + StringBuilder sb = new StringBuilder(); + sb.append(Tag.class.getSimpleName()).append('{'); + sb.append("Key=").append(key); + sb.append(",Vaule=").append(value); + return sb.append('}').toString(); + } + + @JacksonXmlProperty(localName = "Key") + public String key; + + @JacksonXmlProperty(localName = "Value") + public String value; + } + } +} +// CHECKSTYLE:ON diff --git a/core/backend/src/com/pspace/backend/lifecycle/LifecycleFilter.java b/core/backend/src/com/pspace/backend/lifecycle/LifecycleFilter.java new file mode 100644 index 00000000..57dc6458 --- /dev/null +++ b/core/backend/src/com/pspace/backend/lifecycle/LifecycleFilter.java @@ -0,0 +1,419 @@ +/* +* Copyright 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE.md for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.Lifecycle; + +import java.sql.SQLException; +import java.text.SimpleDateFormat; +import java.time.Instant; +import java.time.LocalDateTime; +import java.util.ArrayList; +import java.util.Calendar; +import java.util.Date; +import java.util.TimeZone; + +import org.apache.commons.lang3.StringUtils; +import org.apache.commons.lang3.math.NumberUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.pspace.ifs.ksan.objmanager.LifeCycle; +import com.pspace.ifs.ksan.objmanager.ObjManagerUtil; +import com.pspace.s3format.LifecycleConfiguration; +import com.pspace.s3format.LifecycleConfiguration.Rule; +import com.pspace.s3format.Tagging; + +public class LifecycleFilter { + private final SimpleDateFormat StringDateFormat = new SimpleDateFormat("yyyy-MM-dd"); + private final ObjManagerUtil ObjManager; + private final Logger logger; + + public LifecycleFilter(ObjManagerUtil DB) { + this.ObjManager = DB; + logger = LoggerFactory.getLogger(LifecycleFilter.class); + } + + public boolean Filtering() throws SQLException { + logger.info("Lifecycle Filtering Start!"); + var result = false; + + var BucketList = ObjManager.getBucketList(); + var LifecycleManager = ObjManager.getLifeCycleManagmentInsatance(); + + if (BucketList.size() > 0) { + for (var Bucket : BucketList) { + var BucketName = Bucket.getName(); + var EventList = new ArrayList(); + + // 버킷의 수명주기 설정을 가져온다. + var Lifecycle = GetLifecycleConfiguration(Bucket.getLifecycle()); + + // 버킷의 수명주기 설정을 불러오지 못할 경우 스킵 + if (Lifecycle == null) continue; + + // 버킷의 수명주기 규칙이 비어있거나 재대로 불러오지 못할 경우 스킵 + if (Lifecycle.rules == null || Lifecycle.rules.size() == 0) { + logger.error("[{}] is not a valid rules"); + continue; + } + + // 오브젝트 목록 가져오기 + var ObjectList = ObjManager.listObjects(BucketName, "", 1000); + // 룰정보 가져오기 + var Rules = Lifecycle.rules; + + for (var Rule : Rules) { + // 버킷의 수명주기 설정이 활성화 되어있지 않을 경우 스킵 + if (!isEnabled(Rule.status)) + continue; + + // 오브젝트의 수명주기가 설정되었을 경우 + if (Rule.expiration != null) { + // 오브젝트의 수명주기가 특정한 날짜일 경우 + if (Rule.expiration.date != null) { + // 문자열에서 DATE로 변경 + Date ExpiredTime = StringToDate(Rule.expiration.date); + + // 변경 성공시에만 동작 + if (ExpiredTime != null) { + // 버킷에 존재하는 모든 오브젝트에 대한 수명주기 검사 + for (int Index = ObjectList.size() - 1; Index >= 0; Index--) { + var Object = ObjectList.get(Index); + + // 스킵 체크 + if (LifecycleSkipCheck(Rule, Object)) + continue; + + // 오브젝트의 수명주기가 만료되었을 경우 + if (ExpiredCheck(ExpiredTime)) { + // DB에 저장 + EventList.add(new LifeCycle(0, Object.getBucket(), Object.getPath(), + Object.getVersionId(), "", "")); + + // 목록에서 제거 + ObjectList.remove(Index); + + // Filter로 1개 이상 이벤트를 처리함을 표시 + if (!result) + result = true; + } + } + } + } + // 오브젝트의 수명주기가 일정 기간일 경우 + else if (Rule.expiration.days != null) { + // 기간을 숫자로 변환 + var ExpiredDays = NumberUtils.toInt(Rule.expiration.days); + + // 버킷에 존재하는 모든 오브젝트에 대한 수명주기 검사 + for (int Index = ObjectList.size() - 1; Index >= 0; Index--) { + var Object = ObjectList.get(Index); + + // 스킵 체크 + if (LifecycleSkipCheck(Rule, Object)) + continue; + + // 오브젝트의 수명주기가 만료되었을 경우 + if (ExpiredCheck(Object.getLastModified(), ExpiredDays)) { + // DB에 저장 + EventList.add(new LifeCycle(0, Object.getBucket(), Object.getPath(), + Object.getVersionId(), "", "")); + + // 목록에서 제거 + ObjectList.remove(Index); + + // Filter로 1개 이상 이벤트를 처리함을 표시 + if (!result) + result = true; + } + } + } + } + // 오브젝트의 버저닝 수명주기가 설정 되었을 경우 + if (Rule.versionexpiration != null && !StringUtils.isBlank(Rule.versionexpiration.NoncurrentDays)) { + // 기간을 숫자로 변환 + var ExpiredDays = NumberUtils.toInt(Rule.versionexpiration.NoncurrentDays); + + // 버킷에 존재하는 모든 오브젝트에 대한 수명주기 검사 + for (int Index = ObjectList.size() - 1; Index >= 0; Index--) { + var Object = ObjectList.get(Index); + + // 스킵 체크 + if (LifecycleVersioningSkipCheck(Rule, Object)) + continue; + + // 오브젝트의 수명주기가 만료되었을 경우 + if (ExpiredCheck(Object.getLastModified(), ExpiredDays)) { + // DB에 저장 + EventList.add(new LifeCycle(0, Object.getBucket(), Object.getPath(), + Object.getVersionId(), "", "")); + + // 목록에서 제거 + ObjectList.remove(Index); + + // Filter로 1개 이상 이벤트를 처리함을 표시 + if (!result) + result = true; + } + } + } + + // Multipart의 part 수명주기가 설정 되었을 경우 + if (Rule.abortincompletemultipartupload != null + && !StringUtils.isBlank(Rule.abortincompletemultipartupload.DaysAfterInitiation)) { + // 기간을 숫자로 변환 + var ExpiredDays = NumberUtils.toInt(Rule.abortincompletemultipartupload.DaysAfterInitiation); + + // 업로드 중인 Multipart 목록을 가져온다. + var Multiparts = ObjManager.getMultipartInsatance(BucketName); + for (var Multipart : Multiparts.listUploads("", "", "", "", 1000)) { + + // 오브젝트의 이름이 필터 설정에 만족하지 못할 경우 스킵 + if (LifecycleMultipartSkipCheck(Rule, Multipart.getKey())) + continue; + + // 오브젝트의 Part 목록을 가져온다. + var Parts = Multiparts.getParts(Multipart.getUploadId()); + + for (var Index : Parts.keySet()) { + // 파트를 가져온다. + var Part = Parts.get(Index); + + // Multipart의 수명주기가 만료 되었을 경우 + if (ExpiredCheck(Part.getLastModified(), ExpiredDays)) { + EventList.add(new LifeCycle(0, Multipart.getBucket(), Multipart.getKey(), "", Multipart.getUploadId(), "")); + + // Filter로 1개 이상 이벤트를 처리함을 표시 + if (!result) + result = true; + } + } + } + } + } + + // 이벤트를 DB에 저장 + LifecycleManager.putLifeCycleEvents(EventList); + } + } + return result; + } + + private boolean LifecycleSkipCheck(Rule Rule, com.pspace.ifs.ksan.objmanager.Metadata Object) { + // 마지막 버전이 아닐 경우 스킵 + if (!Object.getLastVersion()) + return true; + + // DeleteMarker일 경우 스킵 + if (isMARKER(Object.getDeleteMarker())) + return true; + + // 필터가 존재할 경우 + if (Rule.filter != null) { + var Tags = getTagging(Object.getTag()); + // And 필터가 존재할 경우 + if (Rule.filter.and != null) { + // Prefix가 설정되어 있을 경우 + if (!StringUtils.isBlank(Rule.filter.and.prefix)) { + // Prefix가 일치하지 않을 경우 스킵 + if (!Object.getPath().startsWith(Rule.filter.and.prefix)) + return true; + } + + // 태그 필터가 설정되어 있을 경우 + if (Rule.filter.and.tag.size() > 0) { + // 오브젝트의 모든 태그를 비교 + int TagCount = Rule.filter.and.tag.size(); + for (var FilterTag : Rule.filter.and.tag) { + for (var ObjectTag : Tags.tagset.tags) { + if (FilterTag.key == ObjectTag.key && FilterTag.value == ObjectTag.value) + TagCount--; + } + } + // 필터에 설정된 태그목록이 오브젝트의 태그목록에 포함되지 않을경우 스킵 + if (TagCount > 0) + return true; + } + + // 최소 크기 필터가 설정되어 있을 경우 + if (!StringUtils.isBlank(Rule.filter.objectSizeGreaterThan)) { + // 오브젝트가 최소크기보다 작을 경우 스킵 + var MinFileSize = NumberUtils.toInt(Rule.filter.objectSizeGreaterThan); + if (Object.getSize() < MinFileSize) + return true; + } + + // 최대 크기 필터가 설정되어 있을 경우 + if (!StringUtils.isBlank(Rule.filter.objectSizeLessThan)) { + // 오브젝트가 최대크기보다 클 경우 스킵 + var MaxFileSize = NumberUtils.toInt(Rule.filter.objectSizeLessThan); + if (Object.getSize() > MaxFileSize) + return true; + } + } + // Prefix가 설정되어 있을 경우 + else if (!StringUtils.isBlank(Rule.filter.prefix)) { + if (!Object.getPath().startsWith(Rule.filter.prefix)) + return true; + } + // 태그 필터가 설정 되어 있을 경우 + else if (Rule.filter.tag != null) { + var FilterTag = Rule.filter.tag; + boolean Find = false; + + // 오브젝트의 모든 태그를 비교 + for (var ObjectTag : Tags.tagset.tags) + if (FilterTag.key == ObjectTag.key && FilterTag.value == ObjectTag.value) + Find = true; + + // 필터에 설정된 태그가 오브젝트의 태그에 존재하지 않을경우 스킵 + if (!Find) + return true; + } + // 최소 크기 필터가 설정되어 있을 경우 + else if (!StringUtils.isBlank(Rule.filter.objectSizeGreaterThan)) { + // 오브젝트가 최소크기보다 작을 경우 스킵 + var MinFileSize = NumberUtils.toInt(Rule.filter.objectSizeGreaterThan); + if (Object.getSize() < MinFileSize) + return true; + } + // 최대 크기 필터가 설정되어 있을 경우 + else if (!StringUtils.isBlank(Rule.filter.objectSizeLessThan)) { + // 오브젝트가 최대크기보다 클 경우 스킵 + var MaxFileSize = NumberUtils.toInt(Rule.filter.objectSizeLessThan); + if (Object.getSize() > MaxFileSize) + return true; + } + } + return false; + } + + private boolean LifecycleVersioningSkipCheck(Rule Rule, com.pspace.ifs.ksan.objmanager.Metadata Object) { + if (Object.getLastVersion()) { + // 마지막 버전이 DeleteMarker 일 경우 + if (isMARKER(Object.getDeleteMarker())) { + // DeleteMarker 옵션이 없을 경우 스킵 + if (Rule.expiration == null) + return true; + if (Rule.expiration.ExpiredObjectDeleteMarker == null) + return true; + else if (!Boolean.parseBoolean(Rule.expiration.ExpiredObjectDeleteMarker)) + return true; + } + // 마지막 버전일 경우 스킵 + else + return true; + } + + // 필터가 존재할 경우 + if (Rule.filter != null && !StringUtils.isBlank(Rule.filter.prefix)) { + // 필터가 일치하지 않을 경우 스킵 + if (!Object.getPath().startsWith(Rule.filter.prefix)) + return true; + } + return false; + } + + private boolean LifecycleMultipartSkipCheck(Rule Rule, String KeyName) { + // 필터가 존재할 경우 + if (Rule.filter != null || !StringUtils.isBlank(Rule.filter.prefix)) { + // 필터가 일치하지 않을 경우 스킵 + if (!KeyName.startsWith(Rule.filter.prefix)) + return true; + } + return false; + } + + /*************************************** + * Utility + *******************************************/ + private final String ENABLED = "Enabled"; + // private final String DISABLED = "Disabled"; + private final String MARKER = "mark"; + + private boolean isEnabled(String Status) { + return ENABLED.equalsIgnoreCase(Status); + } + + // private boolean isDisabled(String Status) {return + // DISABLED.equalsIgnoreCase(Status);} + private boolean isMARKER(String DeleteMarker) { + return MARKER.equalsIgnoreCase(DeleteMarker); + } + + private boolean ExpiredCheck(Date ExpiredTime) { + if (new Date().getTime() > ExpiredTime.getTime()) + return true; + return false; + } + + private boolean ExpiredCheck(long LastModified, int ExpiredDay) { + var LocalDate = LocalDateTime.ofInstant(Instant.ofEpochSecond((Long.MAX_VALUE - LastModified) / 1000000000L), + TimeZone.getDefault().toZoneId()); + + // 마지막 수정시간에서 수명주기가 설정된 시간만큼 날짜를 더하기 + var ExpiredTime = LocalDate.plusDays(ExpiredDay); + var NowTime = LocalDateTime.now(); + + // 만료시간이 지났을 경우 + if (NowTime.isAfter(ExpiredTime)) + return true; + return false; + } + + private boolean ExpiredCheck(Date LastModified, int ExpiredDay) { + // 마지막 수정시간에서 수명주기가 설정된 시간만큼 날짜를 더하기 + Calendar cal = Calendar.getInstance(); + cal.setTime(LastModified); + cal.add(Calendar.DATE, ExpiredDay); + + var ExpiredTime = cal.getTime(); + var NowTime = new Date(); + + // 만료시간이 지났을 경우 + if (NowTime.getTime() > ExpiredTime.getTime()) + return true; + return false; + } + + private Date StringToDate(String Data) { + try { + int Index = Data.indexOf("T"); + return StringDateFormat.parse(Data.substring(0, Index)); + } catch (Exception e) { + logger.error("", e); + return null; + } + } + + private LifecycleConfiguration GetLifecycleConfiguration(String StrLifecycle) { + if (StringUtils.isBlank(StrLifecycle)) return null; + try { + // 수명주기 설정 언마샬링 + return new XmlMapper().readValue(StrLifecycle, LifecycleConfiguration.class); + } catch (Exception e) { + logger.error("Lifecycle read failed \n", e); + return null; + } + } + + private Tagging getTagging(String StrTags) + { + if (StringUtils.isBlank(StrTags)) return new Tagging(); + try { + // 수명주기 설정 언마샬링 + return new XmlMapper().readValue(StrTags, Tagging.class); + } catch (Exception e) { + logger.error("Tag read Failed : {}", e); + return new Tagging(); + } + } +} diff --git a/core/backend/src/com/pspace/backend/lifecycle/LifecycleSender.java b/core/backend/src/com/pspace/backend/lifecycle/LifecycleSender.java new file mode 100644 index 00000000..067ba9e7 --- /dev/null +++ b/core/backend/src/com/pspace/backend/lifecycle/LifecycleSender.java @@ -0,0 +1,143 @@ +/* +* Copyright 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE.md for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.Lifecycle; + +import com.amazonaws.auth.AWSStaticCredentialsProvider; +import com.amazonaws.auth.BasicAWSCredentials; +import com.amazonaws.client.builder.AwsClientBuilder; +import com.amazonaws.services.s3.AmazonS3; +import com.amazonaws.services.s3.AmazonS3ClientBuilder; +import com.amazonaws.services.s3.model.AbortMultipartUploadRequest; +import com.amazonaws.services.s3.model.DeleteObjectRequest; +import com.amazonaws.services.s3.model.DeleteVersionRequest; +import com.pspace.backend.Data.BackendHeaders; +import com.pspace.ifs.ksan.objmanager.ObjManagerUtil; + +import java.sql.SQLException; +import java.util.Date; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class LifecycleSender { + private final Logger logger; + private final AmazonS3 Client; + private final ObjManagerUtil ObjManager; + + public LifecycleSender(ObjManagerUtil ObjManager, String S3URL, String AccessKey, String SecretKey) { + this.ObjManager = ObjManager; + logger = LoggerFactory.getLogger(LifecycleSender.class); + Client = CreateClient(S3URL, AccessKey, SecretKey); + } + + public void Start() { + var LifecycleManager = ObjManager.getLifeCycleManagmentInsatance(); + try { + while (true) { + var EventList = LifecycleManager.getLifeCycleEventList(); + + for (var Event : EventList) { + logger.debug(Event.toString()); + // 결과값 초기화 + String Result = ""; + + // 3회 시도 + for (int i = 0; i < 3; i++) { + // UploadId가 존재하지 않을 경우 오브젝트 삭제 + if (Event.getUploadId() == null || Event.getUploadId().isBlank()) { + // VersionId가 존재하지 않을 경우 일반적인 삭제로 취급 + if (Event.getVersionId() == null || Event.getVersionId().isBlank()) + Result = DeleteObject(Event.getBucketName(), Event.getKey()); + + // VersionId가 존재할 경우 버전아이디를 포함한 삭제로 취급 + else + Result = DeleteObjectVersion(Event.getBucketName(), Event.getKey(), Event.getVersionId()); + } + // UploadId가 존재할 경우 Multipart 삭제 + else + Result = AbortMultipartUpload(Event.getBucketName(), Event.getKey(), Event.getUploadId()); + + // 성공했을 경우 종료 + if (Result.equals("")) + break; + } + // 반환값이 비어있지 않을 경우 - 에러가 발생할 경우 + if (!Result.isBlank()) { + Event.setInDate(new Date()); + Event.setLog(Result); + LifecycleManager.putFailedLifeCycleEvent(Event); + } + LifecycleManager.removeLifeCycleEvent(Event); + } + // DB에서 가져온 목록이 1000개 이하일경우 + if (EventList.size() < 1000) + break; + } + } catch (SQLException e) { + logger.error("", e); + } + + } + + /*************************************** + * Utility + *******************************************/ + + protected AmazonS3 CreateClient(String S3URL, String AccessKey, String SecretKey) { + BasicAWSCredentials awsCreds = new BasicAWSCredentials(AccessKey, SecretKey); + + return AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(awsCreds)) + .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(S3URL, "")) + .withPathStyleAccessEnabled(true).build(); + } + + protected String DeleteObject(String BucketName, String ObjectName) { + var Result = ""; + try { + var Request = new DeleteObjectRequest(BucketName, ObjectName); + Request.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + Request.putCustomRequestHeader(BackendHeaders.HEADER_LIFECYCLE, BackendHeaders.HEADER_DATA); + Client.deleteObject(Request); + } catch (Exception e) { + logger.error("", e); + Result = e.getMessage(); + } + return Result; + } + + protected String DeleteObjectVersion(String BucketName, String ObjectName, String VersionId) { + var Result = ""; + try { + var Request = new DeleteVersionRequest(BucketName, ObjectName, VersionId); + Request.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + Request.putCustomRequestHeader(BackendHeaders.HEADER_LIFECYCLE, BackendHeaders.HEADER_DATA); + Client.deleteVersion(Request); + } catch (Exception e) { + logger.error("", e); + Result = e.getMessage(); + } + return Result; + } + + protected String AbortMultipartUpload(String BucketName, String ObjectName, String UploadId) { + var Result = ""; + try { + var Request = new AbortMultipartUploadRequest(BucketName, ObjectName, UploadId); + Request.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + Request.putCustomRequestHeader(BackendHeaders.HEADER_LIFECYCLE, BackendHeaders.HEADER_DATA); + Client.abortMultipartUpload(Request); + } catch (Exception e) { + logger.error("", e); + Result = e.getMessage(); + } + return Result; + } +} diff --git a/core/backend/src/com/pspace/backend/lifecycle/Main.java b/core/backend/src/com/pspace/backend/lifecycle/Main.java new file mode 100644 index 00000000..f05af249 --- /dev/null +++ b/core/backend/src/com/pspace/backend/lifecycle/Main.java @@ -0,0 +1,135 @@ +/* +* Copyright 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE.md for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.backend.Lifecycle; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.Ksan.KsanConfig; +import com.pspace.Ksan.PortalManager; +import com.pspace.Utility.Util; +import com.pspace.backend.Data.LifecycleConstants; +import com.pspace.backend.Heartbeat.Heartbeat; +import com.pspace.backend.Lifecycle.LifecycleFilter; +import com.pspace.backend.Lifecycle.LifecycleSender; +import com.pspace.ifs.ksan.objmanager.ObjManagerConfig; +import com.pspace.ifs.ksan.objmanager.ObjManagerUtil; + +public class Main { + static final Logger logger = LoggerFactory.getLogger(Main.class); + + public static void main(String[] args) { + logger.info("Lifecycle Manager Start!"); + + // Read Configuration + var KsanConfig = new KsanConfig(LifecycleConstants.AGENT_CONF_PATH); + if (!KsanConfig.GetConfig()) { + logger.error("Config Read Failed!"); + return; + } + + // Get Service Id + var ServiceId = Util.ReadServiceId(LifecycleConstants.LIFECYCLE_SERVICE_ID_PATH); + if (ServiceId == null) { + logger.error("Service Id is Empty"); + return; + } + + // Heartbeat + Thread HBThread; + Heartbeat HB; + try { + HB = new Heartbeat(ServiceId, KsanConfig.MQHost, KsanConfig.MQPort, KsanConfig.MQUser, + KsanConfig.MQPassword); + HBThread = new Thread(() -> HB.Start(KsanConfig.ServiceMonitorInterval)); + HBThread.start(); + } catch (Exception e) { + logger.error("", e); + return; + } + + // Create PortalManager + var Portal = new PortalManager(KsanConfig.PortalHost, KsanConfig.PortalPort, KsanConfig.APIKey); + var today = Util.GetNowDay(); + var AlreadyRun = false; + + while (true) { + try { + if (today != Util.GetNowDay()) + { + today = Util.GetNowDay(); + AlreadyRun = false; + } + + // Read Configuration to Portal + var LifecycleConfig = Portal.GetConfig2Lifecycle(); + if (LifecycleConfig == null) { + logger.error("Lifecycle Config Read Failed!"); + return; + } + logger.info(LifecycleConfig.toString()); + + // Read Region to Portal + var Region = Portal.GetRegion(LifecycleConfig.Region); + if (Region == null) { + logger.error("Region Read Failed!"); + return; + } + + // Schedule + var Schedule = Util.String2Time(LifecycleConfig.Schedule); + if (Schedule < 0) { + logger.error("Schedule is not a valid value. {}\n", LifecycleConfig.Schedule); + return; + } + Thread.sleep(LifecycleConfig.CheckInterval); + + if (!Util.isRun(Schedule) || AlreadyRun) { + continue; + } + AlreadyRun = true; + + // Create ObjManager + ObjManagerUtil ObjManager = null; + try { + ObjManagerConfig ObjConfig = new ObjManagerConfig(); + ObjConfig.dbRepository = LifecycleConfig.DBType; + ObjConfig.dbHost = LifecycleConfig.Host; + ObjConfig.dbport = LifecycleConfig.Port; + ObjConfig.dbName = LifecycleConfig.DatabaseName; + ObjConfig.dbUsername = LifecycleConfig.User; + ObjConfig.dbPassword = LifecycleConfig.Password; + + logger.info(ObjConfig.toString()); + + ObjManager = new ObjManagerUtil(ObjConfig); + } catch (Exception e) { + logger.error("", e); + return; + } + + logger.info("Lifecycle Filter Start!"); + var Filter = new LifecycleFilter(ObjManager); + + if (Filter.Filtering()) { + logger.info("Lifecycle Sender Start!"); + var Sender = new LifecycleSender(ObjManager, Region.GetURL(), Region.AccessKey, Region.SecretKey); + Sender.Start(); + } else + logger.info("Lifecycle filtering Empty!"); + + logger.info("Lifecycle Manager End!"); + } catch (Exception e) { + logger.error("", e); + } + } + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/Main.java b/core/backend/src/com/pspace/backend/logManager/Main.java new file mode 100644 index 00000000..151ad75c --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/Main.java @@ -0,0 +1,106 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +import java.util.TimeZone; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Utility; +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Heartbeat.Heartbeat; +import com.pspace.backend.libs.Ksan.Data.AgentConfig; + +import config.ConfigManager; +import db.DBManager; +import logging.MainLogger; + +public class Main { + static final Logger logger = LoggerFactory.getLogger(Main.class); + + public static void main(String[] args) { + TimeZone.setDefault(TimeZone.getTimeZone("KST")); + logger.info("logManager Start!"); + + // KSAN 설정을 읽어온다. + var ksanConfig = AgentConfig.getInstance(); + if (!ksanConfig.getConfig()) { + logger.error("Config Read Failed!"); + return; + } + logger.info("ksanAgent Read end"); + + // Get Service Id + var ServiceId = Utility.ReadServiceId(Constants.LOGMANAGER_SERVICE_ID_PATH); + if (ServiceId == null) { + logger.error("Service Id is Empty"); + return; + } + // Heartbeat + Thread HBThread; + Heartbeat HB; + try { + HB = new Heartbeat(ServiceId, ksanConfig.MQHost, ksanConfig.MQPort, ksanConfig.MQUser, ksanConfig.MQPassword); + HBThread = new Thread(() -> HB.Start(ksanConfig.ServiceMonitorInterval)); + HBThread.start(); + } catch (Exception e) { + logger.error("", e); + return; + } + + // // 포탈 초기화 + // var portal = PortalManager.getInstance(); + // logger.info("Portal initialized"); + + // LogManager 설정을 읽어온다. + var config = ConfigManager.getInstance(); + try { + config.update(); + } catch (Exception e) { + logger.error("", e); + return; + } + logger.info(config.toString()); + + // // ObjManager 초기화 + // var ObjManager = ObjManagerHelper.getInstance(); + // try { + // ObjManager.init(config.getObjManagerConfig()); + // } catch (Exception e) { + // logger.error("", e); + // return; + // } + // logger.info("objManager initialized"); + + // DB 초기화 + var DB = DBManager.getInstance(); + try{ + DB.init(config.getDBConfig()); + DB.connect(); + }catch (Exception e) + { + logger.error("db connect error : ", e); + return; + } + logger.info("DB initialized"); + + var Logger = new MainLogger(); + if (!Logger.Start(config.getDBPoolSize())) { + logger.error("MainLogger is not started!"); + return; + } + logger.info("Logger Initialization!"); + + while (true) { + Utility.Delay(10000); + } + } +} \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/logManager/config/ConfigManager.java b/core/backend/src/com/pspace/backend/logManager/config/ConfigManager.java new file mode 100644 index 00000000..9a231ee2 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/config/ConfigManager.java @@ -0,0 +1,96 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package config; + +import java.io.IOException; + +import com.pspace.backend.libs.Config.LogManagerConfig; +import com.pspace.backend.libs.Ksan.PortalManager; +import com.pspace.ifs.ksan.objmanager.ObjManagerConfig; + +import db.DBConfig; +import metering.MeteringConfig; + +public class ConfigManager { + + private static LogManagerConfig config = null; + private static PortalManager portal = null; + + public static ConfigManager getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static final ConfigManager INSTANCE = new ConfigManager(); + } + + private ConfigManager() { + portal = PortalManager.getInstance(); + } + + public void update() throws IllegalStateException { + config = portal.getLogManagerConfig(); + if (config == null) + throw new IllegalStateException("Backend Config is not initialized"); + } + + public ObjManagerConfig getObjManagerConfig() throws IOException { + var item = new ObjManagerConfig(); + item.dbRepository = config.DBType; + item.dbHost = config.DBHost; + item.dbport = config.DBPort; + item.dbName = config.DBName; + item.dbUsername = config.DBUser; + item.dbPassword = config.DBPassword; + + return item; + } + + public String getRegion() { + return config.Region; + } + + public int getDBPoolSize() { + return config.DBPoolSize; + } + + public int getDBExpires() { + return config.DBExpires; + } + + public int getCheckInterval() { + return config.CheckInterval; + } + + public DBConfig getDBConfig() { + var dbconfig = new DBConfig(); + dbconfig.Type = config.DBType; + dbconfig.Host = config.DBHost; + dbconfig.Port = config.DBPort; + dbconfig.DatabaseName = config.DBName; + dbconfig.User = config.DBUser; + dbconfig.Password = config.DBPassword; + dbconfig.PoolSize = config.DBPoolSize; + dbconfig.Expires = config.DBExpires; + + return dbconfig; + } + + public MeteringConfig getMeteringConfig() + { + return new MeteringConfig(config.MeterMinute, config.AssertHour); + } + + @Override + public String toString() { + return config.toString(); + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/config/LoggingConfig.java b/core/backend/src/com/pspace/backend/logManager/config/LoggingConfig.java new file mode 100644 index 00000000..c966ee87 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/config/LoggingConfig.java @@ -0,0 +1,55 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package config; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class LoggingConfig { + + private static final int DEFAULT_TIME = 5; + private static final int MILLISECONDS = 60000; + private int Delay; + + public LoggingConfig(int Delay) { + SetDelay(Delay); + } + + public void Init() { + // 기본값 5분 + SetDelay(0); + } + + public void SetDelay(int Minutes) { + if (Minutes > 0) + this.Delay = toMillisec(Minutes); + else + this.Delay = toMillisec(DEFAULT_TIME); + } + + public int getDelay() { + return this.Delay; + } + + public static int toMillisec(int Minutes) { + return Minutes * MILLISECONDS; + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/db/DBConfig.java b/core/backend/src/com/pspace/backend/logManager/db/DBConfig.java new file mode 100644 index 00000000..8df3608c --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/DBConfig.java @@ -0,0 +1,63 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class DBConfig { + public String Type; + public String Host; + public int Port; + public String DatabaseName; + public String MeteringDBName; + public String User; + public String Password; + public int PoolSize; + public int Expires; + + public DBConfig() { + Init(); + } + + public DBConfig(String Host, int Port, String DatabaseName, String MeteringDBName, String User, String Password, int PoolSize, int Expires) { + this.Host = Host; + this.Port = Port; + this.DatabaseName = DatabaseName; + this.MeteringDBName = MeteringDBName; + this.User = User; + this.Password = Password; + this.PoolSize = PoolSize; + if (this.PoolSize > 1) this.PoolSize = 100; + this.Expires = Expires; + } + + public void Init() { + Host = ""; + Port = 3306; + DatabaseName = "event_log"; + MeteringDBName = "event_log"; + User = "root"; + Password = "qwe123"; + PoolSize = 100; + Expires = 0; + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/db/DBManager.java b/core/backend/src/com/pspace/backend/logManager/db/DBManager.java new file mode 100644 index 00000000..06b98ca8 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/DBManager.java @@ -0,0 +1,76 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Data.Lifecycle.LifecycleLogData; +import com.pspace.backend.libs.Data.Replication.ReplicationLogData; +import com.pspace.backend.libs.Data.S3.S3LogData; + +import db.mariaDB.MariaDBManager; +import db.mongoDB.MongoDBManager; + +public class DBManager { + static final Logger logger = LoggerFactory.getLogger(DBManager.class); + IDBManager dbManager; + DBConfig config; + + public static DBManager getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static final DBManager INSTANCE = new DBManager(); + } + + public void init(DBConfig config) { + this.config = config; + if (config.Type.equalsIgnoreCase(Constants.DB_TYPE_MARIADB)) + dbManager = new MariaDBManager(config); + else + dbManager = new MongoDBManager(config); + } + + public void connect() throws Exception { + dbManager.connect(); + } + + public boolean insertLogging(S3LogData data) { + return dbManager.insertLogging(data); + } + + public boolean insertReplicationLog(ReplicationLogData data) { + return dbManager.insertReplicationLog(data); + } + + public boolean insertLifecycleLog(LifecycleLogData data) { + return dbManager.insertLifecycleLog(data); + } + + public boolean insertApiMeter(int minutes) { + return dbManager.insertApiMeter(minutes); + } + + public boolean insertIoMeter(int minutes) { + return dbManager.insertIoMeter(minutes); + } + + public boolean insertApiAsset() { + return dbManager.insertApiAsset(); + } + + public boolean insertIoAsset() { + return dbManager.insertIoAsset(); + } +} \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/logManager/db/IDBManager.java b/core/backend/src/com/pspace/backend/logManager/db/IDBManager.java new file mode 100644 index 00000000..b2618ad7 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/IDBManager.java @@ -0,0 +1,34 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Data.Lifecycle.LifecycleLogData; +import com.pspace.backend.libs.Data.Replication.ReplicationLogData; +import com.pspace.backend.libs.Data.S3.S3LogData; + +public interface IDBManager { + static final Logger logger = LoggerFactory.getLogger(DBManager.class); + + void connect() throws Exception; + + boolean insertLogging(S3LogData Event); + boolean insertReplicationLog(ReplicationLogData data); + boolean insertLifecycleLog(LifecycleLogData data); + + boolean insertApiMeter(int minutes); + boolean insertIoMeter(int minutes); + + boolean insertApiAsset(); + boolean insertIoAsset(); +} \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/logManager/db/mariaDB/LogPreparedStatement.java b/core/backend/src/com/pspace/backend/logManager/db/mariaDB/LogPreparedStatement.java new file mode 100644 index 00000000..4a1dc9f1 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/mariaDB/LogPreparedStatement.java @@ -0,0 +1,569 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db.mariaDB; + +import java.sql.Array; +import java.sql.Blob; +import java.sql.Clob; +import java.sql.Connection; +import java.sql.NClob; +import java.sql.PreparedStatement; +import java.sql.Ref; +import java.sql.SQLException; +import java.sql.SQLWarning; +import java.sql.SQLXML; +import java.sql.Time; +import java.sql.Timestamp; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.StringTokenizer; +import java.sql.ParameterMetaData; +import java.io.InputStream; +import java.io.Reader; +import java.net.URL; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.RowId; + +public class LogPreparedStatement implements java.sql.PreparedStatement { + + private List ParameterValues; + private String SqlTemplate; + private PreparedStatement LogStatement; + + public LogPreparedStatement(Connection connection, String sql) throws SQLException { + LogStatement = connection.prepareStatement(sql); + SqlTemplate = sql; + ParameterValues = new ArrayList(); + } + + public LogPreparedStatement(Connection connection, StringBuffer sql) throws SQLException { + this(connection, sql.toString()); + } + + public void addBatch() throws SQLException { + LogStatement.addBatch(); + } + + public void addBatch(String sql) throws SQLException { + LogStatement.addBatch(sql); + } + + public void cancel() throws SQLException { + LogStatement.cancel(); + } + + public void clearBatch() throws SQLException { + LogStatement.clearBatch(); + } + + public void clearParameters() throws SQLException { + LogStatement.clearParameters(); + } + + public void clearWarnings() throws SQLException { + LogStatement.clearWarnings(); + } + + public void close() throws SQLException { + LogStatement.close(); + } + + public boolean execute() throws SQLException { + return LogStatement.execute(); + } + + public boolean execute(String sql) throws SQLException { + return LogStatement.execute(sql); + } + + public boolean execute(String sql, String[] strArray) throws SQLException { + return LogStatement.execute(sql, strArray); + } + + public boolean execute(String sql, int[] intArray) throws SQLException { + return LogStatement.execute(sql, intArray); + } + + public boolean execute(String sql, int intValue) throws SQLException { + return LogStatement.execute(sql, intValue); + } + + public int[] executeBatch() throws SQLException { + return LogStatement.executeBatch(); + } + + public ResultSet executeQuery() throws SQLException { + return LogStatement.executeQuery(); + } + + public ResultSet executeQuery(String sql) throws SQLException { + return LogStatement.executeQuery(sql); + } + + public int executeUpdate() throws SQLException { + return LogStatement.executeUpdate(); + } + + public int executeUpdate(String sql) throws SQLException { + return LogStatement.executeUpdate(sql); + } + + public int executeUpdate(String sql, String[] strList) throws SQLException { + return LogStatement.executeUpdate(sql, strList); + } + + public int executeUpdate(String sql, int[] intList) throws SQLException { + return LogStatement.executeUpdate(sql, intList); + } + + public int executeUpdate(String sql, int intValue) throws SQLException { + return LogStatement.executeUpdate(sql, intValue); + } + + public Connection getConnection() throws SQLException { + return LogStatement.getConnection(); + } + + public int getFetchDirection() throws SQLException { + return LogStatement.getFetchDirection(); + } + + public int getFetchSize() throws SQLException { + return LogStatement.getFetchSize(); + } + + public int getMaxFieldSize() throws SQLException { + return LogStatement.getMaxFieldSize(); + } + + public ResultSet getGeneratedKeys() throws SQLException { + return LogStatement.getGeneratedKeys(); + } + + public int getMaxRows() throws SQLException { + return LogStatement.getMaxRows(); + } + + public ResultSetMetaData getMetaData() throws SQLException { + return LogStatement.getMetaData(); + } + + public boolean getMoreResults() throws SQLException { + return LogStatement.getMoreResults(); + } + + public boolean getMoreResults(int Value) throws SQLException { + return LogStatement.getMoreResults(Value); + } + + public int getQueryTimeout() throws SQLException { + return LogStatement.getQueryTimeout(); + } + + public ResultSet getResultSet() throws SQLException { + return LogStatement.getResultSet(); + } + + public int getResultSetConcurrency() throws SQLException { + return LogStatement.getResultSetConcurrency(); + } + + public int getResultSetType() throws SQLException { + return LogStatement.getResultSetType(); + } + + public int getUpdateCount() throws SQLException { + return LogStatement.getUpdateCount(); + } + + public SQLWarning getWarnings() throws SQLException { + return LogStatement.getWarnings(); + } + + public void setArray(int i, Array x) throws SQLException { + LogStatement.setArray(i, x); + saveQueryParamValue(i, x); + } + + public void setAsciiStream(int ParameterIndex, InputStream x, int Length) throws SQLException { + LogStatement.setAsciiStream(ParameterIndex, x, Length); + saveQueryParamValue(ParameterIndex, x); + } + + public void setBigDecimal(int ParameterIndex, java.math.BigDecimal x) throws SQLException { + LogStatement.setBigDecimal(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setURL(int _int, URL uRL) throws SQLException { + LogStatement.setURL(_int, uRL); + } + + public void setBinaryStream(int ParameterIndex, InputStream x, int Length) throws SQLException { + LogStatement.setBinaryStream(ParameterIndex, x, Length); + saveQueryParamValue(ParameterIndex, x); + } + + public void setBlob(int i, Blob x) throws SQLException { + LogStatement.setBlob(i, x); + saveQueryParamValue(i, x); + } + + public void setBoolean(int ParameterIndex, boolean x) throws SQLException { + LogStatement.setBoolean(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setByte(int ParameterIndex, byte x) throws SQLException { + LogStatement.setByte(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setBytes(int ParameterIndex, byte[] x) throws SQLException { + LogStatement.setBytes(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setCharacterStream(int ParameterIndex, java.io.Reader reader, int Length) throws SQLException { + LogStatement.setCharacterStream(ParameterIndex, reader, Length); + saveQueryParamValue(ParameterIndex, reader); + } + + public void setClob(int i, Clob x) throws SQLException { + LogStatement.setClob(i, x); + saveQueryParamValue(i, x); + } + + public void setCursorName(String name) throws SQLException { + LogStatement.setCursorName(name); + } + + public void setDate(int ParameterIndex, java.sql.Date x) throws SQLException { + LogStatement.setDate(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setDate(int ParameterIndex, java.sql.Date x, java.util.Calendar cal) throws SQLException { + LogStatement.setDate(ParameterIndex, x, cal); + saveQueryParamValue(ParameterIndex, x); + } + + public void setDouble(int ParameterIndex, double x) throws SQLException { + LogStatement.setDouble(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setEscapeProcessing(boolean enable) throws SQLException { + LogStatement.setEscapeProcessing(enable); + } + + public void setFetchDirection(int direction) throws SQLException { + LogStatement.setFetchDirection(direction); + } + + public void setFetchSize(int rows) throws SQLException { + LogStatement.setFetchSize(rows); + } + + public void setFloat(int ParameterIndex, float x) throws SQLException { + LogStatement.setFloat(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + + } + + public void setInt(int ParameterIndex, int x) throws SQLException { + LogStatement.setInt(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setLong(int ParameterIndex, long x) throws SQLException { + LogStatement.setLong(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setMaxFieldSize(int max) throws SQLException { + LogStatement.setMaxFieldSize(max); + } + + public void setMaxRows(int max) throws SQLException { + LogStatement.setMaxRows(max); + } + + public void setNull(int ParameterIndex, int sqlType) throws SQLException { + LogStatement.setNull(ParameterIndex, sqlType); + saveQueryParamValue(ParameterIndex, null); + } + + public void setNull(int paramIndex, int sqlType, String typeName) throws SQLException { + LogStatement.setNull(paramIndex, sqlType, typeName); + saveQueryParamValue(paramIndex, null); + } + + public void setObject(int ParameterIndex, Object x) throws SQLException { + LogStatement.setObject(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setObject(int ParameterIndex, Object x, int targetSqlType) throws SQLException { + LogStatement.setObject(ParameterIndex, x, targetSqlType); + saveQueryParamValue(ParameterIndex, x); + } + + public void setObject(int ParameterIndex, Object x, int targetSqlType, int scale) throws SQLException { + LogStatement.setObject(ParameterIndex, x, targetSqlType, scale); + saveQueryParamValue(ParameterIndex, x); + } + + public void setQueryTimeout(int seconds) throws SQLException { + LogStatement.setQueryTimeout(seconds); + } + + public void setRef(int i, Ref x) throws SQLException { + LogStatement.setRef(i, x); + saveQueryParamValue(i, x); + } + + public void setShort(int ParameterIndex, short x) throws SQLException { + LogStatement.setShort(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setString(int ParameterIndex, String x) throws SQLException { + LogStatement.setString(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setTime(int ParameterIndex, Time x) throws SQLException { + LogStatement.setTime(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setTime(int ParameterIndex, Time x, java.util.Calendar cal) throws SQLException { + LogStatement.setTime(ParameterIndex, x, cal); + saveQueryParamValue(ParameterIndex, x); + } + + public void setTimestamp(int ParameterIndex, Timestamp x) throws SQLException { + LogStatement.setTimestamp(ParameterIndex, x); + saveQueryParamValue(ParameterIndex, x); + } + + public void setTimestamp(int ParameterIndex, Timestamp x, java.util.Calendar cal) + throws SQLException { + LogStatement.setTimestamp(ParameterIndex, x, cal); + saveQueryParamValue(ParameterIndex, x); + } + + public ParameterMetaData getParameterMetaData() throws SQLException { + return LogStatement.getParameterMetaData(); + } + + public int getResultSetHoldability() throws SQLException { + return LogStatement.getResultSetHoldability(); + } + + private void saveQueryParamValue(int position, Object obj) { + String Value; + if (obj instanceof String || obj instanceof Date) { + Value = "'" + obj + "'"; + } else { + if (obj == null) { + Value = "null"; + } else { + Value = obj.toString(); + } + } + + while (position >= ParameterValues.size()) { + ParameterValues.add(null); + } + + ParameterValues.set(position, Value); + } + + public String toString() { + StringBuffer Buf = new StringBuffer(); + int MarkCount = 0; + StringTokenizer tok = new StringTokenizer(SqlTemplate + " ", "?"); + while (tok.hasMoreTokens()) { + String oneChunk = tok.nextToken(); + Buf.append(oneChunk); + + try { + Object Value; + if (ParameterValues.size() > 1 + MarkCount) { + Value = ParameterValues.get(1 + MarkCount++); + } else { + if (tok.hasMoreTokens()) { + Value = null; + } else { + Value = ""; + } + } + Buf.append("" + Value); + } catch (Throwable e) { + Buf.append("Error : " + e.toString()); + } + } + return Buf.toString().trim(); + } + + @Override + public boolean isClosed() throws SQLException { + return LogStatement.isClosed(); + } + + @Override + public void setPoolable(boolean poolable) throws SQLException { + LogStatement.setPoolable(poolable); + } + + @Override + public boolean isPoolable() throws SQLException { + return LogStatement.isPoolable(); + } + + @Override + public void closeOnCompletion() throws SQLException { + LogStatement.closeOnCompletion(); + } + + @Override + public boolean isCloseOnCompletion() throws SQLException { + return LogStatement.isCloseOnCompletion(); + } + + @Override + public T unwrap(Class iface) throws SQLException { + return LogStatement.unwrap(iface); + } + + @Override + public boolean isWrapperFor(Class iface) throws SQLException { + return LogStatement.isWrapperFor(iface); + } + + @Override + public void setRowId(int ParameterIndex, RowId x) throws SQLException { + LogStatement.setRowId(ParameterIndex, x); + + } + + @Override + public void setNString(int ParameterIndex, String Value) throws SQLException { + LogStatement.setNString(ParameterIndex, Value); + + } + + @Override + public void setNCharacterStream(int ParameterIndex, Reader Value, long Length) throws SQLException { + LogStatement.setNCharacterStream(ParameterIndex, Value); + + } + + @Override + public void setNClob(int ParameterIndex, NClob Value) throws SQLException { + LogStatement.setNClob(ParameterIndex, Value); + + } + + @Override + public void setClob(int ParameterIndex, Reader reader, long Length) throws SQLException { + LogStatement.setClob(ParameterIndex, reader, Length); + + } + + @Override + public void setBlob(int ParameterIndex, InputStream inputStream, long Length) throws SQLException { + LogStatement.setBlob(ParameterIndex, inputStream, Length); + + } + + @Override + public void setNClob(int ParameterIndex, Reader reader, long Length) throws SQLException { + LogStatement.setNClob(ParameterIndex, reader, Length); + + } + + @Override + public void setSQLXML(int ParameterIndex, SQLXML xmlObject) throws SQLException { + LogStatement.setSQLXML(ParameterIndex, xmlObject); + + } + + @Override + public void setAsciiStream(int ParameterIndex, InputStream x, long Length) throws SQLException { + LogStatement.setAsciiStream(ParameterIndex, x, Length); + + } + + @Override + public void setBinaryStream(int ParameterIndex, InputStream x, long Length) throws SQLException { + LogStatement.setBinaryStream(ParameterIndex, x, Length); + + } + + @Override + public void setCharacterStream(int ParameterIndex, Reader reader, long Length) throws SQLException { + LogStatement.setCharacterStream(ParameterIndex, reader, Length); + + } + + @Override + public void setAsciiStream(int ParameterIndex, InputStream x) throws SQLException { + LogStatement.setAsciiStream(ParameterIndex, x); + + } + + @Override + public void setBinaryStream(int ParameterIndex, InputStream x) throws SQLException { + LogStatement.setBinaryStream(ParameterIndex, x); + + } + + @Override + public void setCharacterStream(int ParameterIndex, Reader reader) throws SQLException { + LogStatement.setCharacterStream(ParameterIndex, reader); + + } + + @Override + public void setNCharacterStream(int ParameterIndex, Reader Value) throws SQLException { + LogStatement.setNCharacterStream(ParameterIndex, Value); + } + + @Override + public void setClob(int ParameterIndex, Reader reader) throws SQLException { + LogStatement.setClob(ParameterIndex, reader); + + } + + @Override + public void setBlob(int ParameterIndex, InputStream inputStream) throws SQLException { + LogStatement.setBlob(ParameterIndex, inputStream); + + } + + @Override + public void setNClob(int ParameterIndex, Reader reader) throws SQLException { + LogStatement.setNClob(ParameterIndex, reader); + + } + + @Override + @Deprecated(since = "1.2") + public void setUnicodeStream(int ParameterIndex, InputStream x, int Length) throws SQLException { + LogStatement.setUnicodeStream(ParameterIndex, x, Length); + } + +} diff --git a/core/backend/src/com/pspace/backend/logManager/db/mariaDB/MariaDBManager.java b/core/backend/src/com/pspace/backend/logManager/db/mariaDB/MariaDBManager.java new file mode 100644 index 00000000..fa1045e6 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/mariaDB/MariaDBManager.java @@ -0,0 +1,207 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db.mariaDB; + +import java.sql.SQLException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Data.Lifecycle.LifecycleLogData; +import com.pspace.backend.libs.Data.Replication.ReplicationLogData; +import com.pspace.backend.libs.Data.S3.S3LogData; +import com.zaxxer.hikari.HikariConfig; +import com.zaxxer.hikari.HikariDataSource; + +import db.DBConfig; +import db.IDBManager; +import db.table.Lifecycle.LifecycleLogQuery; +import db.table.Logging.LoggingQuery; +import db.table.Metering.ApiMeteringQuery; +import db.table.Metering.IoMeteringQuery; +import db.table.replication.ReplicationLogQuery; + +public class MariaDBManager implements IDBManager { + static final Logger logger = LoggerFactory.getLogger(MariaDBManager.class); + + HikariDataSource ds = null; + + DBConfig config; + + public MariaDBManager(DBConfig config) { + this.config = config; + } + + public void connect() throws Exception { + var URL = String.format( + "jdbc:mariadb://%s:%d/%s?createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=utf8mb4", + config.Host, config.Port, config.DatabaseName); + var hikariConfig = new HikariConfig(); + hikariConfig.setJdbcUrl(URL); + hikariConfig.setUsername(config.User); + hikariConfig.setPassword(config.Password); + hikariConfig.setDriverClassName("org.mariadb.jdbc.Driver"); + hikariConfig.setConnectionTestQuery("select 1"); + hikariConfig.addDataSourceProperty("maxPoolSize", config.PoolSize); + hikariConfig.addDataSourceProperty("minPoolSize", config.PoolSize); + hikariConfig.setPoolName("ksan"); + hikariConfig.setMaximumPoolSize(config.PoolSize); + hikariConfig.setMinimumIdle(config.PoolSize); + ds = new HikariDataSource(hikariConfig); + + createTables(); + } + + /***************************** Select **********************************/ + + ////////////////////// Logging ////////////////////// + public List getLoggingEventList(String BucketName) { + var rmap = Select(LoggingQuery.getSelect(BucketName)); + return LoggingQuery.getListMariaDB(rmap); + } + + /***************************** Insert *****************************/ + + private boolean createTables() { + // DB Create Table + return Execute(ReplicationLogQuery.getCreate()) && + Execute(ApiMeteringQuery.createMeter()) && + Execute(ApiMeteringQuery.createAsset()) && + Execute(IoMeteringQuery.createMeter()) && + Execute(IoMeteringQuery.createAsset()); + } + + @Override + public boolean insertLogging(S3LogData data) { + return insert(LoggingQuery.getInsert(), LoggingQuery.getInsertParameters(data)); + } + + @Override + public boolean insertReplicationLog(ReplicationLogData data) { + return insert(ReplicationLogQuery.getInsert(), ReplicationLogQuery.getInsertDBParameters(data)); + } + + @Override + public boolean insertLifecycleLog(LifecycleLogData data) { + return insert(LifecycleLogQuery.getInsert(), LifecycleLogQuery.getInsertDBParameters(data)); + } + + @Override + public boolean insertApiMeter(int minutes) { + return Execute(ApiMeteringQuery.insertMeter(minutes)); + } + + @Override + public boolean insertIoMeter(int minutes) { + return Execute(IoMeteringQuery.getInsertMeter(minutes)); + } + + @Override + public boolean insertApiAsset() { + return Execute(ApiMeteringQuery.insertAsset()); + } + + @Override + public boolean insertIoAsset() { + return Execute(IoMeteringQuery.getInsertAsset()); + } + + /***************************** Expiration *****************************/ + public boolean Expiration() { + if (!Delete(ReplicationLogQuery.getExpiration(config.Expires))) + return false; + return true; + } + + /*********************** Utility ***********************/ + private boolean Execute(String Query) { + try ( + var conn = ds.getConnection(); + var stmt = new LogPreparedStatement(conn, Query);) { + logger.debug(stmt.toString()); + stmt.execute(); + return true; + } catch (SQLException e) { + logger.error("Query Error : {}", Query, e); + } catch (Exception e) { + logger.error("Query Error : {}", Query, e); + } + return false; + } + + private boolean insert(String Query, List Params) { + try ( + var conn = ds.getConnection(); + var stmt = new LogPreparedStatement(conn, Query);) { + int index = 1; + for (Object Param : Params) + stmt.setObject(index++, Param); + logger.debug(stmt.toString()); + return stmt.executeUpdate() > 0; + } catch (SQLException e) { + logger.error("Query Error : {}", Query, e); + } catch (Exception e) { + logger.error("Query Error : {}", Query, e); + } + return false; + } + + private List> Select(String Query) { + try ( + var conn = ds.getConnection(); + var stmt = new LogPreparedStatement(conn, Query); + var rs = stmt.executeQuery();) { + // logger.debug(stmt.toString()); + + var rmap = new ArrayList>(); + + var md = rs.getMetaData(); + int columns = md.getColumnCount(); + + while (rs.next()) { + + HashMap map = null; + map = new HashMap(columns); + for (int i = 1; i <= columns; ++i) { + map.put(md.getColumnName(i), rs.getObject(i)); + } + rmap.add(map); + } + return rmap; + + } catch (SQLException e) { + logger.error("Query Error : {}", Query, e); + } catch (Exception e) { + logger.error("Query Error : {}", Query, e); + } + return null; + } + + private boolean Delete(String Query) { + try ( + var conn = ds.getConnection(); + var stmt = new LogPreparedStatement(conn, Query);) { + stmt.execute(); + logger.debug(stmt.toString()); + stmt.close(); + conn.close(); + return true; + } catch (SQLException e) { + logger.error("Query Error : {}", Query, e); + } catch (Exception e) { + logger.error("Query Error : {}", Query, e); + } + return false; + } +} \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/logManager/db/mongoDB/MongoDBManager.java b/core/backend/src/com/pspace/backend/logManager/db/mongoDB/MongoDBManager.java new file mode 100644 index 00000000..991f1ad7 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/mongoDB/MongoDBManager.java @@ -0,0 +1,157 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db.mongoDB; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; + +import org.bson.Document; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.mongodb.MongoClientSettings; +import com.mongodb.MongoCredential; +import com.mongodb.ServerAddress; +import com.mongodb.client.MongoClient; +import com.mongodb.client.MongoClients; +import com.mongodb.client.MongoDatabase; +import com.pspace.backend.libs.Data.Lifecycle.LifecycleLogData; +import com.pspace.backend.libs.Data.Replication.ReplicationLogData; +import com.pspace.backend.libs.Data.S3.S3LogData; + +import db.DBConfig; +import db.IDBManager; +import db.table.Lifecycle.LifecycleLogQuery; +import db.table.Logging.LoggingQuery; +import db.table.replication.ReplicationLogQuery; + +public class MongoDBManager implements IDBManager { + static final Logger logger = LoggerFactory.getLogger(MongoDBManager.class); + + MongoClient mongo; + MongoDatabase db; + DBConfig config; + + public MongoDBManager(DBConfig Config) { + this.config = Config; + } + + public void connect() throws Exception { + var credential = MongoCredential.createCredential(config.User, config.DatabaseName, + config.Password.toCharArray()); + var serverAddress = new ServerAddress(config.Host, config.Port); + mongo = MongoClients.create(MongoClientSettings.builder() + .applyToClusterSettings(builder -> builder.hosts(Arrays.asList(serverAddress))) + .credential(credential) + .build()); + db = mongo.getDatabase(config.DatabaseName); + } + + /***************************** Select **********************************/ + + ////////////////////// Logging ////////////////////// + public List getLoggingEventList(String BucketName) { + var rmap = Select(LoggingQuery.getSelect(BucketName)); + return LoggingQuery.getListMariaDB(rmap); + } + + /***************************** Insert *****************************/ + + public boolean insertLogging(S3LogData data) { + return Insert(LoggingQuery.DB_TABLE_NAME, LoggingQuery.getInsertDocument(data)); + } + + public boolean insertReplicationLog(ReplicationLogData data) { + return Insert(ReplicationLogQuery.DB_TABLE_NAME, ReplicationLogQuery.getInsertDocument(data)); + } + + public boolean insertLifecycleLog(LifecycleLogData data) { + return Insert(LifecycleLogQuery.DB_TABLE_NAME, LifecycleLogQuery.getInsertDocument(data)); + } + + /***************************** Expiration *****************************/ + public boolean Expiration() { + return true; + } + + /*********************** Utility ***********************/ + private boolean Insert(String tableName, Document document) { + try { + var collection = db.getCollection(tableName); + collection.insertOne(document); + } catch (Exception e) { + logger.error("Error : {}", tableName, e); + } + return false; + } + + private List> Select(String tableName) { + try { + var rmap = new ArrayList>(); + + var collection = db.getCollection(tableName); + var cursor = collection.find().iterator(); + + while (cursor.hasNext()) { + Document item = cursor.next(); + var keys = item.keySet(); + var map = new HashMap(keys.size()); + for (var key : keys) + map.put(key, item.get(key)); + rmap.add(map); + } + cursor.close(); + return rmap; + } catch (Exception e) { + logger.error("Query Error : {}", tableName, e); + } + return null; + } + + @Override + public boolean insertApiMeter(int minutes) { + // TODO Auto-generated method stub + return false; + } + + @Override + public boolean insertIoMeter(int minutes) { + // TODO Auto-generated method stub + return false; + } + + @Override + public boolean insertApiAsset() { + // TODO Auto-generated method stub + return false; + } + + @Override + public boolean insertIoAsset() { + // TODO Auto-generated method stub + return false; + } + + // private boolean Delete(String Query) { + // try { + // stmt.execute(); + // logger.debug(stmt.toString()); + // stmt.close(); + // conn.close(); + // return true; + // } catch (Exception e) { + // logger.error("Query Error : {}", Query, e); + // } + // return false; + // } +} \ No newline at end of file diff --git a/portal/Portal/src/app/shared/pipes/empty-to-nbsp.pipe.ts b/core/backend/src/com/pspace/backend/logManager/db/table/BaseLogQuery.java similarity index 71% rename from portal/Portal/src/app/shared/pipes/empty-to-nbsp.pipe.ts rename to core/backend/src/com/pspace/backend/logManager/db/table/BaseLogQuery.java index 7570de1d..f69763e0 100644 --- a/portal/Portal/src/app/shared/pipes/empty-to-nbsp.pipe.ts +++ b/core/backend/src/com/pspace/backend/logManager/db/table/BaseLogQuery.java @@ -1,22 +1,20 @@ /* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version +* the GNU General Public License as published by the Free Software Foundation, either version * 3 of the License. See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ -import { Pipe, PipeTransform } from '@angular/core'; +package db.table; -@Pipe({ name: 'emptyToNbsp' }) -export class EmptyToNbspPipe implements PipeTransform { - transform(value: any): string { - if (value === 0 || value) { - return value; - } +public interface BaseLogQuery { + final int MAX_QUERY_SIZE = 1000; + final String DB_DATE_FORMAT = "%Y-%m-%d %k:%i:%s"; - return ' '; - } + public String getCreateTableQuery(); + public String getInsertQuery(); + public String getExpiration(int Days); } diff --git a/core/backend/src/com/pspace/backend/logManager/db/table/Lifecycle/LifecycleLogQuery.java b/core/backend/src/com/pspace/backend/logManager/db/table/Lifecycle/LifecycleLogQuery.java new file mode 100644 index 00000000..7be9ed18 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/table/Lifecycle/LifecycleLogQuery.java @@ -0,0 +1,79 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db.table.Lifecycle; + +import java.util.ArrayList; +import java.util.List; + +import org.bson.Document; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Data.Lifecycle.LifecycleLogData; + +import db.table.QueryConstants; + +public class LifecycleLogQuery { + static final Logger logger = LoggerFactory.getLogger(LifecycleLogQuery.class); + + public static final String DB_TABLE_NAME = "LIFECYCLE_LOG"; + public static final String DB_ID = "ID"; + public static final String DB_IN_DATE = "INDATE"; + public static final String DB_BUCKETNAME = "BUCKET_NAME"; + public static final String DB_OBJECTNAME = "OBJECT_NAME"; + public static final String DB_VERSIONID = "VERSION_ID"; + public static final String DB_UPLOADID = "UPLOAD_ID"; + public static final String DB_MESSAGE = "MESSAGE"; + + public static String getCreateTable() { + return "CREATE TABLE IF NOT EXISTS " + DB_TABLE_NAME + " ( " + + DB_ID + " BIGINT AUTO_INCREMENT PRIMARY KEY, " + + DB_IN_DATE + " varchar(64) NOT NULL, " + + DB_BUCKETNAME + " VARCHAR(64) NOT NULL, " + + DB_OBJECTNAME + " VARCHAR(2048) NOT NULL, " + + DB_VERSIONID + " VARCHAR(256) NOT NULL, " + + DB_UPLOADID + " VARCHAR(256) NULL, " + + DB_MESSAGE + " TEXT NULL) " + + "ENGINE=INNODB DEFAULT CHARSET=utf8mb4;"; + } + + public static String getInsert() { + return String.format("INSERT INTO %s(%s, %s, %s, %s, %s) VALUES(?, ?, ?, ?, ?)", + DB_TABLE_NAME, DB_BUCKETNAME, DB_OBJECTNAME, DB_VERSIONID, DB_UPLOADID, DB_MESSAGE); + } + + public static String getExpiration(int Days) { + return String.format("delete FROM %s where %s < date_add(date_format(now() , '%s'), interval -%d day);", + DB_TABLE_NAME, DB_IN_DATE, QueryConstants.DB_DATE_FORMAT, Days); + } + + public static List getInsertDBParameters(LifecycleLogData data) { + var param = new ArrayList(); + param.add(data.BucketName); + param.add(data.ObjectName); + param.add(data.VersionId); + param.add(data.UploadId); + param.add(data.Message); + + return param; + } + + public static Document getInsertDocument(LifecycleLogData data) { + var param = new Document(); + param.put(DB_BUCKETNAME, data.BucketName); + param.put(DB_OBJECTNAME, data.ObjectName); + param.put(DB_VERSIONID, data.VersionId); + param.put(DB_UPLOADID, data.UploadId); + param.put(DB_MESSAGE, data.Message); + + return param; + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/db/table/Logging/LoggingLogQuery.java b/core/backend/src/com/pspace/backend/logManager/db/table/Logging/LoggingLogQuery.java new file mode 100644 index 00000000..b0bd1990 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/table/Logging/LoggingLogQuery.java @@ -0,0 +1,40 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db.table.Logging; + +public class LoggingLogQuery { + + public final String TABLE_NAME = "LOGGING_LOG"; + public final String ID = "ID"; + public final String SOURCE_BUCKET = "SOURCE_BUCKET"; + public final String TARGET_BUCKET = "TARGET_BUCKET"; + public final String TARGET_KEY = "TARGET_KEY"; + public final String DATE_TIME = "DATE_TIME"; + public final String LAST_LOG_ID = "LAST_LOG_ID"; + public final String MESSAGE = "MESSAGE"; + + public String getCreateTableQuery() { + return "CREATE TABLE IF NOT EXISTS " + TABLE_NAME + " ( " + + ID + " bigint auto_increment primary key, " + + SOURCE_BUCKET + " varchar(64) NOT NULL, " + + TARGET_BUCKET + " varchar(64) NOT NULL, " + + TARGET_KEY + " varchar(2048) NOT NULL, " + + DATE_TIME + " timestamp(3) NOT NULL DEFAULT current_timestamp(3), " + + LAST_LOG_ID + " bigint NOT NULL, " + + MESSAGE + " TEXT NULL);"; + } + + public String getInsertQuery() { + return String.format("INSERT INTO %s(%s, %s, %s, %s, %s) VALUES(?, ?, ?, ?, ?);", + TABLE_NAME, SOURCE_BUCKET, TARGET_BUCKET, TARGET_KEY, LAST_LOG_ID, + MESSAGE); + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/db/table/Logging/LoggingQuery.java b/core/backend/src/com/pspace/backend/logManager/db/table/Logging/LoggingQuery.java new file mode 100644 index 00000000..bc5cb300 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/table/Logging/LoggingQuery.java @@ -0,0 +1,205 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db.table.Logging; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; + +import com.pspace.backend.libs.Data.S3.S3LogData; + +import org.bson.Document; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class LoggingQuery { + static final Logger logger = LoggerFactory.getLogger(LoggingQuery.class); + + public static final String DB_TABLE_NAME = "S3LOGGINGS"; + public static final String DB_ID = "ID"; + public static final String DB_USER_NAME = "USER_NAME"; + public static final String DB_BUCKET_NAME = "BUCKET_NAME"; + public static final String DB_IN_DATE = "IN_DATE"; + public static final String DB_REMOTE_HOST = "REMOTE_HOST"; + public static final String DB_REQUEST_USER = "REQUEST_USER"; + public static final String DB_REQUEST_ID = "REQUEST_ID"; + public static final String DB_OPERATION = "OPERATION"; + public static final String DB_OBJECT_NAME = "OBJECT_NAME"; + public static final String DB_REQUEST_URI = "REQUEST_URI"; + public static final String DB_STATUS_CODE = "STATUS_CODE"; + public static final String DB_ERROR_CODE = "ERROR_CODE"; + public static final String DB_RESPONSE_LENGTH = "RESPONSE_LENGTH"; + public static final String DB_OBJECT_LENGTH = "OBJECT_LENGTH"; + public static final String DB_TOTAL_TIME = "TOTAL_TIME"; + public static final String DB_REQUEST_LENGTH = "REQUEST_LENGTH"; + public static final String DB_REFERER = "REFERER"; + public static final String DB_USER_AGENT = "USER_AGENT"; + public static final String DB_VERSION_ID = "VERSION_ID"; + public static final String DB_HOST_ID = "HOST_ID"; + public static final String DB_SIGN = "SIGN"; + public static final String DB_SSL_GROUP = "SSL_GROUP"; + public static final String DB_SIGN_TYPE = "SIGN_TYPE"; + public static final String DB_ENDPOINT = "ENDPOINT"; + public static final String DB_TLS_VERSION = "TLS_VERSION"; + + public static String getCreateTable() { + return "CREATE TABLE IF NOT EXISTS " + DB_TABLE_NAME + " ( " + + DB_ID + " bigint auto_increment primary key, " + + DB_USER_NAME + " varchar(64) DEFAULT NULL, " + + DB_BUCKET_NAME + " varchar(64) DEFAULT NULL, " + + DB_IN_DATE + " varchar(64) NOT NULL, " + + DB_REMOTE_HOST + " varchar(256) DEFAULT NULL, " + + DB_REQUEST_USER + " varchar(64) DEFAULT NULL, " + + DB_REQUEST_ID + " varchar(64) DEFAULT NULL, " + + DB_OPERATION + " varchar(64) DEFAULT NULL, " + + DB_OBJECT_NAME + " varchar(2048) DEFAULT NULL, " + + DB_REQUEST_URI + " varchar(2048) DEFAULT NULL, " + + DB_STATUS_CODE + " int DEFAULT NULL, " + + DB_ERROR_CODE + " varchar(256) DEFAULT NULL, " + + DB_RESPONSE_LENGTH + " bigint DEFAULT NULL, " + + DB_OBJECT_LENGTH + " bigint DEFAULT NULL, " + + DB_TOTAL_TIME + " bigint DEFAULT NULL, " + + DB_REQUEST_LENGTH + " bigint DEFAULT NULL, " + + DB_REFERER + " varchar(64) DEFAULT NULL, " + + DB_USER_AGENT + " varchar(256) DEFAULT NULL, " + + DB_VERSION_ID + " varchar(64) DEFAULT NULL, " + + DB_HOST_ID + " varchar(256) DEFAULT NULL, " + + DB_SIGN + " varchar(32) DEFAULT NULL, " + + DB_SSL_GROUP + " varchar(64) DEFAULT NULL, " + + DB_SIGN_TYPE + " varchar(32) DEFAULT NULL, " + + DB_ENDPOINT + " varchar(64) DEFAULT NULL, " + + DB_TLS_VERSION + " varchar(32) DEFAULT NULL) " + + "ENGINE=INNODB DEFAULT CHARSET=utf8mb4;"; + } + + public static String getInsert() { + return String.format( + "INSERT INTO %s(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)" + + " VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);", + DB_TABLE_NAME, + DB_USER_NAME, DB_BUCKET_NAME, DB_IN_DATE, DB_REMOTE_HOST, DB_REQUEST_USER, DB_REQUEST_ID, + DB_OPERATION, DB_OBJECT_NAME, DB_REQUEST_URI, DB_STATUS_CODE, DB_ERROR_CODE, DB_RESPONSE_LENGTH, + DB_OBJECT_LENGTH, DB_TOTAL_TIME, DB_REQUEST_LENGTH, DB_REFERER, DB_USER_AGENT, DB_VERSION_ID, + DB_HOST_ID, DB_SIGN, DB_SSL_GROUP, DB_SIGN_TYPE, DB_ENDPOINT, DB_TLS_VERSION); + } + + public static String getSelect(String BucketName) { + return String.format("SELECT * FROM %s WHERE %s = '%s';", DB_TABLE_NAME, DB_BUCKET_NAME, BucketName); + } + + public static String getDelete(String BucketName, long Index) { + return String.format("DELETE FROM %s WHERE %s = '%s' and %s <= %d;", DB_TABLE_NAME, DB_BUCKET_NAME, BucketName, + DB_ID, Index); + } + + public static String getClear(String BucketName) { + return String.format("DELETE FROM %s WHERE %s = '%s';", DB_TABLE_NAME, DB_BUCKET_NAME, BucketName); + } + + public static List getListMariaDB(List> resultList) { + if (resultList == null) + return null; + var MyList = new ArrayList(); + try { + for (HashMap result : resultList) { + MyList.add(new S3LogData( + (String) result.get(DB_USER_NAME), + (String) result.get(DB_BUCKET_NAME), + (String) result.get(DB_IN_DATE), + (String) result.get(DB_REMOTE_HOST), + (String) result.get(DB_REQUEST_USER), + (String) result.get(DB_REQUEST_ID), + (String) result.get(DB_OPERATION), + (String) result.get(DB_OBJECT_NAME), + (String) result.get(DB_REQUEST_URI), + (int) result.get(DB_STATUS_CODE), + (String) result.get(DB_ERROR_CODE), + (long) result.get(DB_RESPONSE_LENGTH), + (long) result.get(DB_OBJECT_LENGTH), + (long) result.get(DB_TOTAL_TIME), + (long) result.get(DB_REQUEST_LENGTH), + (String) result.get(DB_REFERER), + (String) result.get(DB_USER_AGENT), + (String) result.get(DB_VERSION_ID), + (String) result.get(DB_HOST_ID), + (String) result.get(DB_SIGN), + (String) result.get(DB_SSL_GROUP), + (String) result.get(DB_SIGN_TYPE), + (String) result.get(DB_ENDPOINT), + (String) result.get(DB_TLS_VERSION))); + } + } catch (Exception e) { + logger.error("", e); + return null; + } + return MyList; + } + + public static List getInsertParameters(S3LogData data) { + var param = new ArrayList(); + param.add(data.UserName); + param.add(data.BucketName); + param.add(data.Date); + param.add(data.RemoteHost); + param.add(data.RequestUser); + param.add(data.RequestId); + param.add(data.Operation); + param.add(data.ObjectName); + param.add(data.RequestURI); + param.add(data.StatusCode); + param.add(data.ErrorCode); + param.add(data.ResponseLength); + param.add(data.ObjectLength); + param.add(data.TotalTime); + param.add(data.RequestLength); + param.add(data.Referer); + param.add(data.UserAgent); + param.add(data.VersionId); + param.add(data.HostId); + param.add(data.Sign); + param.add(data.SSLGroup); + param.add(data.SignType); + param.add(data.EndPoint); + param.add(data.TLSVersion); + + return param; + } + + public static Document getInsertDocument(S3LogData data) { + var param = new Document(); + param.put(DB_USER_NAME, data.UserName); + param.put(DB_BUCKET_NAME, data.BucketName); + param.put(DB_IN_DATE, data.Date); + param.put(DB_REMOTE_HOST, data.RemoteHost); + param.put(DB_REQUEST_USER, data.RequestUser); + param.put(DB_REQUEST_ID, data.RequestId); + param.put(DB_OPERATION, data.Operation); + param.put(DB_OBJECT_NAME, data.ObjectName); + param.put(DB_REQUEST_URI, data.RequestURI); + param.put(DB_STATUS_CODE, data.StatusCode); + param.put(DB_ERROR_CODE, data.ErrorCode); + param.put(DB_RESPONSE_LENGTH, data.ResponseLength); + param.put(DB_OBJECT_LENGTH, data.ObjectLength); + param.put(DB_TOTAL_TIME, data.TotalTime); + param.put(DB_REQUEST_LENGTH, data.RequestLength); + param.put(DB_REFERER, data.Referer); + param.put(DB_USER_AGENT, data.UserAgent); + param.put(DB_VERSION_ID, data.VersionId); + param.put(DB_HOST_ID, data.HostId); + param.put(DB_SIGN, data.Sign); + param.put(DB_SSL_GROUP, data.SSLGroup); + param.put(DB_SIGN_TYPE, data.SignType); + param.put(DB_ENDPOINT, data.EndPoint); + param.put(DB_TLS_VERSION, data.TLSVersion); + + return param; + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/db/table/Metering/ApiMeteringQuery.java b/core/backend/src/com/pspace/backend/logManager/db/table/Metering/ApiMeteringQuery.java new file mode 100644 index 00000000..1c6f1ba0 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/table/Metering/ApiMeteringQuery.java @@ -0,0 +1,59 @@ +package db.table.Metering; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import db.table.Logging.LoggingQuery; + +public class ApiMeteringQuery { + static final Logger log = LoggerFactory.getLogger(ApiMeteringQuery.class); + + static final String DB_TABLE_NAME_METER = "BUCKET_API_METERS"; + static final String DB_TABLE_NAME_ASSET = "BUCKET_API_ASSETS"; + static final String DB_INDATE = "INDATE"; + static final String DB_USER = "USER_NAME"; + static final String DB_BUCKET = "BUCKET_NAME"; + static final String DB_EVENT = "OPERATION"; + static final String DB_COUNT = "COUNT"; + + public static String createMeter() { + return "CREATE TABLE IF NOT EXISTS " + DB_TABLE_NAME_METER + " ( " + + DB_INDATE + " datetime NOT NULL, " + + DB_USER + " varchar(64) NOT NULL, " + + DB_BUCKET + " varchar(64) NOT NULL, " + + DB_EVENT + " varchar(200) NOT NULL, " + + DB_COUNT + " bigint DEFAULT NULL, " + + "PRIMARY KEY (" + DB_INDATE + ", " + DB_USER + ", " + DB_BUCKET + ", " + DB_EVENT + "));"; + } + + public static String createAsset() { + return "CREATE TABLE IF NOT EXISTS " + DB_TABLE_NAME_ASSET + " ( " + + DB_INDATE + " datetime NOT NULL, " + + DB_USER + " varchar(64) NOT NULL, " + + DB_BUCKET + " varchar(64) NOT NULL, " + + DB_EVENT + " varchar(200) NOT NULL, " + + DB_COUNT + " bigint DEFAULT NULL, " + + "PRIMARY KEY (" + DB_INDATE + ", " + DB_USER + ", " + DB_BUCKET + ", " + DB_EVENT + "));"; + } + + public static String insertMeter(int times) { + String.format("insert into %s (%s, %s, %s, %s, %s)", DB_TABLE_NAME_METER, DB_INDATE, DB_USER, DB_BUCKET, DB_EVENT, DB_COUNT); + + return "insert into " + DB_TABLE_NAME_METER + "("+ DB_INDATE + ", " + DB_USER + ", " + DB_BUCKET + ", " + DB_EVENT + ", " + DB_COUNT + ")" + + "select now(), user_name, bucket_name, operation, count(*) as count " + + "from (select * from " + LoggingQuery.DB_TABLE_NAME + + " where DATE_SUB(NOW(), INTERVAL "+ times +" MINUTE) < date_time) as s3logging, " + // + S3BucketQuery.DB_TABLE_NAME + + " where s3logging.bucket_name = bucketlist.bucket group by user_name, bucket_name, operation;"; + } + + public static String insertAsset() { + return "insert into " + DB_TABLE_NAME_ASSET + "(INDATE, USER, BUCKET, EVENT, COUNT)" + + " select now(), user, bucket, event, sum(count) from " + DB_TABLE_NAME_METER + + " group by user, bucket, event;"; + } + + public static String meterClear() { + return String.format("TRUNCATE %s;", DB_TABLE_NAME_METER); + } +} \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/logManager/db/table/Metering/IoMeteringQuery.java b/core/backend/src/com/pspace/backend/logManager/db/table/Metering/IoMeteringQuery.java new file mode 100644 index 00000000..37c6a4a4 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/table/Metering/IoMeteringQuery.java @@ -0,0 +1,58 @@ +package db.table.Metering; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import db.table.Logging.LoggingQuery; + +public class IoMeteringQuery { + static final Logger log = LoggerFactory.getLogger(IoMeteringQuery.class); + + static final String DB_TABLE_NAME_METER = "BUCKET_IO_METER"; + static final String DB_TABLE_NAME_ASSET = "BUCKET_IO_ASSET"; + static final String DB_INDATE = "INDATE"; + static final String DB_USER = "USER"; + static final String DB_BUCKET = "BUCKET"; + static final String DB_UPLOAD = "UPLOAD"; + static final String DB_DOWNLOAD = "DOWNLOAD"; + + public static String createMeter() { + return "CREATE TABLE IF NOT EXISTS " + DB_TABLE_NAME_METER + " ( " + + DB_INDATE + " datetime NOT NULL, " + + DB_USER + " varchar(64) NOT NULL, " + + DB_BUCKET + " varchar(64) NOT NULL, " + + DB_UPLOAD + " bigint DEFAULT NULL, " + + DB_DOWNLOAD + " bigint DEFAULT NULL, " + + "PRIMARY KEY (" + DB_INDATE + ", " + DB_USER + ", " + DB_BUCKET + "));"; + } + + public static String createAsset() { + return "CREATE TABLE IF NOT EXISTS " + DB_TABLE_NAME_ASSET + " ( " + + DB_INDATE + " datetime NOT NULL, " + + DB_USER + " varchar(64) NOT NULL, " + + DB_BUCKET + " varchar(64) NOT NULL, " + + DB_UPLOAD + " bigint DEFAULT NULL, " + + DB_DOWNLOAD + " bigint DEFAULT NULL, " + + "PRIMARY KEY (" + DB_INDATE + ", " + DB_USER + ", " + DB_BUCKET + "));"; + } + + public static String getInsertMeter(int times) { + return "insert into " + DB_TABLE_NAME_METER + "(indate, volume, user, bucket, upload, download) " + + "select now(), user_name, bucketlist.volume, bucket_name, sum(request_length), sum(response_length) " + + "from (select * from " + LoggingQuery.DB_TABLE_NAME + + " where DATE_SUB(NOW(), INTERVAL " + times + " MINUTE) < date_time) as s3logging, " + // + S3BucketQuery.DB_TABLE_NAME + + " where s3logging.bucket_name = bucketlist.bucket group by user_name, bucket_name;"; + } + + public static String getInsertAsset() { + return "insert into " + DB_TABLE_NAME_ASSET + "(indate, volume, user, bucket, upload, download)" + + " select now(), volume, user, bucket, sum(upload), sum(download) from " + + DB_TABLE_NAME_METER + + " group by volume, user, bucket;"; + } + + public static String getMeterClear() { + return String.format("TRUNCATE %s;", DB_TABLE_NAME_METER); + } +} \ No newline at end of file diff --git a/core/common/Enums/EnumResponseResult.py b/core/backend/src/com/pspace/backend/logManager/db/table/QueryConstants.java similarity index 80% rename from core/common/Enums/EnumResponseResult.py rename to core/backend/src/com/pspace/backend/logManager/db/table/QueryConstants.java index 59396a87..2911a70e 100644 --- a/core/common/Enums/EnumResponseResult.py +++ b/core/backend/src/com/pspace/backend/logManager/db/table/QueryConstants.java @@ -1,4 +1,4 @@ -""" +/* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version @@ -7,13 +7,10 @@ * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -from enum import Enum - - -class EnumResponseResult(Enum): - Error = -1 - Warning = 0 - Success = 1 +*/ +package db.table; +public class QueryConstants { + public static final int MAX_QUERY_SIZE = 1000; + public static final String DB_DATE_FORMAT = "%Y-%m-%d %k:%i:%s"; +} diff --git a/core/backend/src/com/pspace/backend/logManager/db/table/replication/ReplicationLogQuery.java b/core/backend/src/com/pspace/backend/logManager/db/table/replication/ReplicationLogQuery.java new file mode 100644 index 00000000..be2a0089 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/db/table/replication/ReplicationLogQuery.java @@ -0,0 +1,87 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package db.table.replication; + +import java.util.ArrayList; +import java.util.List; + +import org.bson.Document; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Data.Replication.ReplicationLogData; + +import db.table.QueryConstants; + +public class ReplicationLogQuery { + static final Logger logger = LoggerFactory.getLogger(ReplicationLogQuery.class); + + public static final String DB_TABLE_NAME = "REPLICATION_LOG"; + public static final String DB_ID = "ID"; + public static final String DB_IN_DATE = "INDATE"; + public static final String DB_OPERATION = "OPERATION"; + public static final String DB_OBJECTNAME = "OBJECT_NAME"; + public static final String DB_VERSIONID = "VERSION_ID"; + public static final String DB_SOURCEBUCKETNAME = "SOURCE_BUCKET_NAME"; + public static final String DB_TARGETBUCKETNAME = "TARGET_BUCKET_NAME"; + public static final String DB_TARGETREGION = "TARGET_REGION"; + public static final String DB_MESSAGE = "MESSAGE"; + + public static String getCreate() { + return "CREATE TABLE IF NOT EXISTS " + DB_TABLE_NAME + " ( " + + DB_ID + " BIGINT AUTO_INCREMENT PRIMARY KEY, " + + DB_IN_DATE + " TIMESTAMP DEFAULT CURRENT_TIMESTAMP, " + + DB_OPERATION + " VARCHAR(64) NOT NULL, " + + DB_OBJECTNAME + " VARCHAR(2048) NOT NULL, " + + DB_VERSIONID + " VARCHAR(32) NOT NULL, " + + DB_SOURCEBUCKETNAME + " VARCHAR(256) NOT NULL, " + + DB_TARGETBUCKETNAME + " VARCHAR(256) NOT NULL," + + DB_TARGETREGION + " VARCHAR(32) NULL, " + + DB_MESSAGE + " TEXT NULL) " + + "ENGINE=INNODB DEFAULT CHARSET=utf8mb4;"; + } + + public static String getInsert() { + return String.format("INSERT INTO %s(%s, %s, %s, %s, %s, %s, %s) VALUES(?, ?, ?, ?, ?, ?, ?)", + DB_TABLE_NAME, DB_OPERATION, DB_OBJECTNAME, DB_VERSIONID, DB_SOURCEBUCKETNAME, DB_TARGETBUCKETNAME, DB_TARGETREGION, DB_MESSAGE); + } + + public static String getExpiration(int Days) { + return String.format("delete FROM %s where %s < date_add(date_format(now() , '%s'), interval -%d day);", + DB_TABLE_NAME, DB_IN_DATE, QueryConstants.DB_DATE_FORMAT, Days); + } + + public static List getInsertDBParameters(ReplicationLogData data) { + var param = new ArrayList(); + param.add(data.Operation); + param.add(data.ObjectName); + param.add(data.VersionId); + param.add(data.SourceBucketName); + param.add(data.TargetBucketName); + param.add(data.TargetRegion); + param.add(data.Message); + + return param; + } + + public static Document getInsertDocument(ReplicationLogData data) { + var param = new Document(); + param.put(DB_OPERATION, data.Operation); + param.put(DB_OBJECTNAME, data.ObjectName); + param.put(DB_VERSIONID, data.VersionId); + param.put(DB_SOURCEBUCKETNAME, data.SourceBucketName); + param.put(DB_TARGETBUCKETNAME, data.TargetBucketName); + param.put(DB_TARGETREGION, data.TargetRegion); + param.put(DB_MESSAGE, data.Message); + + return param; + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/docker/DockerFile b/core/backend/src/com/pspace/backend/logManager/docker/DockerFile new file mode 100644 index 00000000..572205b2 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/docker/DockerFile @@ -0,0 +1,27 @@ +FROM maven:3.8.6-jdk-11 as libsbuild +WORKDIR /src +COPY ./core/src/com/pspace/ifs/ksan/libs/ /src/ +RUN --mount=type=cache,target=/root/.m2 mvn -f /src/pom.xml install +######################################################################## +FROM maven:3.8.6-jdk-11 as objmanagerbuild +WORKDIR /src +COPY ./core/src/com/pspace/ifs/ksan/objmanager/ /src/ +RUN --mount=type=cache,target=/root/.m2 mvn -f /src/pom.xml install +######################################################################## +FROM maven:3.8.6-jdk-11 as backendlibsbuild +WORKDIR /src +COPY ./core/backend/src/com/pspace/backend/libs/ /src/ +RUN --mount=type=cache,target=/root/.m2 mvn -f /src/pom.xml install +####################################################################### +FROM maven:3.8.6-jdk-11 as build +WORKDIR /src +COPY ./core/backend/src/com/pspace/backend/logManager/ /src/ +RUN --mount=type=cache,target=/root/.m2 mvn -f /src/pom.xml package +######################################################################## +FROM openjdk:11-jre-slim +WORKDIR /app +RUN mkdir -p /usr/local/ksan/etc/ +COPY ./core/backend/src/com/pspace/backend/logManager/ksanLogManager_log_conf.xml /usr/local/ksan/etc/ +COPY --from=build /src/target/ksanLogManager.jar /usr/local/ksan/sbin/ +COPY ./core/backend/src/com/pspace/backend/logManager/docker/start.sh /app +CMD ["/app/start.sh"] \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/logManager/docker/start.sh b/core/backend/src/com/pspace/backend/logManager/docker/start.sh new file mode 100644 index 00000000..8af9a31b --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/docker/start.sh @@ -0,0 +1,2 @@ +#!/bin/bash +java -jar -Dlogback.configurationFile=/usr/local/ksan/etc/ksanLogManager_log_conf.xml /usr/local/ksan/sbin/ksanLogManager.jar \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/logManager/ksanLogManager_log_conf.xml b/core/backend/src/com/pspace/backend/logManager/ksanLogManager_log_conf.xml new file mode 100644 index 00000000..c89e1a3f --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/ksanLogManager_log_conf.xml @@ -0,0 +1,23 @@ + + + + + + %-5level %d{yyyy-MM-dd HH:mm:ss.SSS, ${logback.timezone:-Asia/Seoul}} [%thread{5}][%logger{10}.%method:%line] : %msg%n + + + + ${BACKEND_LOG_PATH}/log-manager.log + + ${BACKEND_LOG_PATH}/log-manager.log.%d{yyyy-MM-dd, ${logback.timezone:-Asia/Seoul}}.%i.log.gz + 7 + 100MB + + + %-5level %d{yyyy-MM-dd HH:mm:ss.SSS, ${logback.timezone:-Asia/Seoul}} [%thread{5}][%logger{10}.%method:%line] : %msg%n + + + + + + \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/logManager/logging/LifecycleLogReceiver.java b/core/backend/src/com/pspace/backend/logManager/logging/LifecycleLogReceiver.java new file mode 100644 index 00000000..d7dce9e4 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/logging/LifecycleLogReceiver.java @@ -0,0 +1,61 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package logging; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.core.type.TypeReference; +import com.pspace.ifs.ksan.libs.mq.MQCallback; +import com.pspace.ifs.ksan.libs.mq.MQResponse; +import com.pspace.ifs.ksan.libs.mq.MQResponseCode; +import com.pspace.ifs.ksan.libs.mq.MQResponseType; +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Data.Lifecycle.LifecycleLogData; + +import db.DBManager; + +public class LifecycleLogReceiver implements MQCallback { + private final Logger logger = LoggerFactory.getLogger(LifecycleLogReceiver.class); + private final ObjectMapper Mapper = new ObjectMapper(); + + @Override + public MQResponse call(String routingKey, String body) { + + try { + logger.debug("{} -> {}", routingKey, body); + + if (!routingKey.equals(Constants.MQ_BINDING_LIFECYCLE_LOG)) + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + + // 문자열을 S3LogData 클래스로 변환 + var event = Mapper.readValue(body, new TypeReference() { + }); + // 변환 실패시 + if (event == null) + throw new Exception("Invalid LifecycleLogData : " + body); + + // Get DB + var db = DBManager.getInstance(); + + //DB에 저장 + db.insertLifecycleLog(event); + + } catch (Exception e) { + logger.error("", e); + return new MQResponse(MQResponseType.ERROR, MQResponseCode.MQ_UNKNOWN_ERROR, e.getMessage(), 0); + } + + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + } + +} diff --git a/core/backend/src/com/pspace/backend/logManager/logging/MainLogger.java b/core/backend/src/com/pspace/backend/logManager/logging/MainLogger.java new file mode 100644 index 00000000..fd29fe27 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/logging/MainLogger.java @@ -0,0 +1,99 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package logging; + +import java.util.ArrayList; +import java.util.List; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Ksan.Data.AgentConfig; +import com.pspace.ifs.ksan.libs.mq.MQReceiver; + +import config.ConfigManager; + +public class MainLogger { + final Logger logger = LoggerFactory.getLogger(MainLogger.class); + final ConfigManager Config; + protected final AgentConfig agent; + + List s3LogReceivers = new ArrayList(); + List replicationLogReceivers = new ArrayList(); + List LifecycleLogReceivers = new ArrayList(); + + // Thread SendThread; + // SendLogger Sender; + + public MainLogger() { + this.agent = AgentConfig.getInstance(); + Config = ConfigManager.getInstance(); + } + + public boolean Start(int ThreadCount) { + try { + // Sender = new SendLogger(DB, Config, Region); + // SendThread = new Thread(() -> Sender.Run()); + // SendThread.start(); + for (int index = 0; index < ThreadCount; index++) { + s3LogReceivers.add(new MQReceiver( + agent.MQHost, + agent.MQPort, + agent.MQUser, + agent.MQPassword, + Constants.MQ_QUEUE_LOG_MANAGER_S3_LOG, + Constants.MQ_KSAN_LOG_EXCHANGE, + false, + "", + Constants.MQ_BINDING_GW_LOG, + new S3LogReceiver())); + replicationLogReceivers.add(new MQReceiver( + agent.MQHost, + agent.MQPort, + agent.MQUser, + agent.MQPassword, + Constants.MQ_QUEUE_LOG_MANAGER_REPLICATION_EVENT_LOG, + Constants.MQ_KSAN_LOG_EXCHANGE, + false, + "", + Constants.MQ_BINDING_REPLICATION_LOG, + new ReplicationLogReceiver())); + LifecycleLogReceivers.add(new MQReceiver( + agent.MQHost, + agent.MQPort, + agent.MQUser, + agent.MQPassword, + Constants.MQ_QUEUE_LOG_MANAGER_LIFECYCLE_EVENT_LOG, + Constants.MQ_KSAN_LOG_EXCHANGE, + false, + "", + Constants.MQ_BINDING_LIFECYCLE_LOG, + new LifecycleLogReceiver())); + } + + return true; + } catch (Exception e) { + logger.error("", e); + return false; + } + } + + public void Quit() { + try { + // Sender.Quit(); + // SendThread.join(); + } catch (Exception e) { + logger.error("", e); + } + } + +} diff --git a/core/backend/src/com/pspace/backend/logManager/logging/ReplicationLogReceiver.java b/core/backend/src/com/pspace/backend/logManager/logging/ReplicationLogReceiver.java new file mode 100644 index 00000000..fd19991d --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/logging/ReplicationLogReceiver.java @@ -0,0 +1,61 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package logging; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.core.type.TypeReference; +import com.pspace.ifs.ksan.libs.mq.MQCallback; +import com.pspace.ifs.ksan.libs.mq.MQResponse; +import com.pspace.ifs.ksan.libs.mq.MQResponseCode; +import com.pspace.ifs.ksan.libs.mq.MQResponseType; +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Data.Replication.ReplicationLogData; + +import db.DBManager; + +public class ReplicationLogReceiver implements MQCallback { + private final Logger logger = LoggerFactory.getLogger(ReplicationLogReceiver.class); + private final ObjectMapper Mapper = new ObjectMapper(); + + @Override + public MQResponse call(String routingKey, String body) { + + try { + logger.debug("{} -> {}", routingKey, body); + + if (!routingKey.equals(Constants.MQ_BINDING_REPLICATION_LOG)) + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + + // 문자열을 ReplicationLogData 클래스로 변환 + var event = Mapper.readValue(body, new TypeReference() { + }); + // 변환 실패시 + if (event == null) + throw new Exception("Invalid ReplicationLogData : " + body); + + // Get DB + var db = DBManager.getInstance(); + + //DB에 저장 + db.insertReplicationLog(event); + + } catch (Exception e) { + logger.error("", e); + return new MQResponse(MQResponseType.ERROR, MQResponseCode.MQ_UNKNOWN_ERROR, e.getMessage(), 0); + } + + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + } + +} diff --git a/core/backend/src/com/pspace/backend/logManager/logging/S3LogReceiver.java b/core/backend/src/com/pspace/backend/logManager/logging/S3LogReceiver.java new file mode 100644 index 00000000..8a8e8f39 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/logging/S3LogReceiver.java @@ -0,0 +1,61 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package logging; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.core.type.TypeReference; +import com.pspace.ifs.ksan.libs.mq.MQCallback; +import com.pspace.ifs.ksan.libs.mq.MQResponse; +import com.pspace.ifs.ksan.libs.mq.MQResponseCode; +import com.pspace.ifs.ksan.libs.mq.MQResponseType; +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Data.S3.S3LogData; + +import db.DBManager; + +public class S3LogReceiver implements MQCallback { + private final Logger logger = LoggerFactory.getLogger(S3LogReceiver.class); + private final ObjectMapper Mapper = new ObjectMapper(); + + @Override + public MQResponse call(String routingKey, String body) { + + try { + logger.debug("{} -> {}", routingKey, body); + + if (!routingKey.equals(Constants.MQ_BINDING_GW_LOG)) + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + + // 문자열을 S3LogData 클래스로 변환 + var event = Mapper.readValue(body, new TypeReference() { + }); + // 변환 실패시 + if (event == null) + throw new Exception("Invalid S3LogData : " + body); + + // Get DB + var db = DBManager.getInstance(); + + // DB에 저장 + db.insertLogging(event); + + } catch (Exception e) { + logger.error("", e); + return new MQResponse(MQResponseType.ERROR, MQResponseCode.MQ_UNKNOWN_ERROR, e.getMessage(), 0); + } + + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + } + +} diff --git a/core/backend/src/com/pspace/backend/logManager/logging/SendLogger.java b/core/backend/src/com/pspace/backend/logManager/logging/SendLogger.java new file mode 100644 index 00000000..c6706d67 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/logging/SendLogger.java @@ -0,0 +1,164 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package logging; + +import java.time.LocalDateTime; +import java.time.format.DateTimeFormatter; +import java.util.Random; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.amazonaws.auth.AWSStaticCredentialsProvider; +import com.amazonaws.auth.BasicAWSCredentials; +import com.amazonaws.client.builder.AwsClientBuilder; +import com.amazonaws.services.s3.AmazonS3; +import com.amazonaws.services.s3.AmazonS3ClientBuilder; +import com.amazonaws.services.s3.model.AccessControlList; +import com.amazonaws.services.s3.model.CanonicalGrantee; +import com.amazonaws.services.s3.model.ObjectMetadata; +import com.amazonaws.services.s3.model.Permission; +import com.amazonaws.services.s3.model.PutObjectRequest; +import com.pspace.backend.libs.Utility; +import com.pspace.backend.libs.Data.BackendHeaders; +import com.pspace.backend.libs.Ksan.Data.S3RegionData; +import com.pspace.backend.libs.s3format.LoggingConfiguration; + +import db.DBManager; + +public class SendLogger { + private final char[] TEXT = { 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', + 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9' }; + private Random rand = new Random(); + private final Logger logger = LoggerFactory.getLogger(SendLogger.class); + private final int DEFAULT_UNIQUE_STRING_SIZE = 16; + + // private final LoggingConfig Config; + // private final DBManager DB; + // private final AmazonS3 Client; + private boolean Stop = false; + private boolean Quit = false; + + public SendLogger(S3RegionData Region) { + // this.DB = DB; + // this.Config = Config; + // this.Client = CreateClient(Region); + } + + public void Run() { + // while (!Quit) { + // // 일시정지 확인 + // if (Stop) { + // Utility.Delay(Config.getDelay()); + // continue; + // } + + // try { + // var BucketList = DB.getBucketInfoLoggingList(); + // for (var BucketInfo : BucketList) { + // var SourceBucketName = BucketInfo.BucketName; + // var TargetBucketName = BucketInfo.Loggings.loggingEnabled.targetBucket; + // var Prefix = BucketInfo.Loggings.loggingEnabled.targetPrefix; + // var Grants = getAccessControlList(BucketInfo.Loggings.loggingEnabled.targetGrants); + + // // 로그 출력 + // var Loggings = DB.getLoggingEventList(SourceBucketName); + // var LastIndex = 0L; + // var Data = ""; + // for (var Logging : Loggings) { + // LastIndex = Logging.Index; + // Data += Logging.Print(); + // } + // if (LastIndex > 0) { + // var Key = NewKey(Prefix); + // PutLogging(TargetBucketName, Key, Data, Grants); + // DB.DeleteLoggingEvent(SourceBucketName, LastIndex); + // } + // } + // } catch (Exception e) { + // logger.error("", e); + // } + + // // 대기 + // Utility.Delay(Config.getDelay()); + // } + } + + public void Stop() { + Stop = true; + } + + public void Start() { + Stop = false; + } + + public void Quit() { + Quit = true; + } + + /********************** Utility ****************************/ + + protected AmazonS3 CreateClient(S3RegionData Region) { + BasicAWSCredentials awsCreds = new BasicAWSCredentials(Region.AccessKey, Region.SecretKey); + + return AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(awsCreds)) + .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(Region.getHttpURL(), "")) + .withPathStyleAccessEnabled(true).build(); + } + + protected String RandomText(int Length) { + StringBuffer sb = new StringBuffer(); + + for (int i = 0; i < Length; i++) + sb.append(TEXT[rand.nextInt(TEXT.length)]); + return sb.toString(); + } + + protected String NewKey(String Prefix) { + var now = LocalDateTime.now(); + var formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd-HH-mm-ss"); + if (Prefix == null) Prefix = ""; + return Prefix + now.format(formatter) + RandomText(DEFAULT_UNIQUE_STRING_SIZE); + } + + protected AccessControlList getAccessControlList(LoggingConfiguration.LoggingEnabled.TargetGrants targetGrants) + { + if (targetGrants == null) return null; + if (targetGrants.grants == null) return null; + var Result = new AccessControlList(); + for(var Item : targetGrants.grants) + { + var User = new CanonicalGrantee(Item.grantee.id); + User.setDisplayName(Item.grantee.displayName); + + Result.grantPermission(User, Permission.parsePermission(Item.permission)); + } + return Result; + } + + protected boolean PutLogging(String BucketName, String Key, String Body, AccessControlList Grants) { + try { + var Request = new PutObjectRequest(BucketName, Key, Utility.CreateBody(Body), new ObjectMetadata()); + // 권한 설정 + if (Grants != null) Request.setAccessControlList(Grants); + + //헤더 추가 + Request.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + Request.putCustomRequestHeader(BackendHeaders.HEADER_LOGGING, BackendHeaders.HEADER_DATA); + + // Client.putObject(Request); + return true; + } catch (Exception e) { + logger.error("", e); + return false; + } + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/logging/data/LoggingLogData.java b/core/backend/src/com/pspace/backend/logManager/logging/data/LoggingLogData.java new file mode 100644 index 00000000..bbd8ad87 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/logging/data/LoggingLogData.java @@ -0,0 +1,36 @@ +package logging.data; + +import java.util.ArrayList; +import java.util.List; + +import org.joda.time.DateTime; + +public class LoggingLogData{ + public long Index; + public String SourceBucket; + public String TargetBucket; + public String TargetKey; + public long LastIndex; + public DateTime InDate; + public String ErrorMessage; + + public void Init() { + Index = 0; + SourceBucket = ""; + TargetBucket = ""; + TargetKey = ""; + LastIndex = 0; + InDate = null; + ErrorMessage = ""; + } + + public List getInsertDBParameters() { + var param = new ArrayList(); + param.add(SourceBucket); + param.add(TargetBucket); + param.add(TargetKey); + param.add(LastIndex); + param.add(ErrorMessage); + return param; + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/metering/MainMetering.java b/core/backend/src/com/pspace/backend/logManager/metering/MainMetering.java new file mode 100644 index 00000000..dc1aaf9d --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/metering/MainMetering.java @@ -0,0 +1,56 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package metering; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import db.DBManager; + +public class MainMetering { + private final Logger logger = LoggerFactory.getLogger(MainMetering.class); + private final MeteringConfig Config; + private final DBManager DB; + + private Thread SendThread; + private SendMetering Sender; + + public MainMetering(DBManager DB, MeteringConfig Config) { + this.DB = DB; + this.Config = Config; + } + + public boolean Start() { + try { + Sender = new SendMetering(DB, Config); + SendThread = new Thread(() -> Sender.Run()); + SendThread.start(); + return true; + } catch (Exception e) { + logger.error("", e); + return false; + } + } + + public void Stop() { + Sender.Stop(); + } + + public void Quit() { + Sender.Quit(); + try { + SendThread.join(); + } catch (Exception e) { + logger.error("", e); + } + } + +} diff --git a/core/backend/src/com/pspace/backend/logManager/metering/MeteringConfig.java b/core/backend/src/com/pspace/backend/logManager/metering/MeteringConfig.java new file mode 100644 index 00000000..0e812839 --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/metering/MeteringConfig.java @@ -0,0 +1,57 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package metering; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +public class MeteringConfig { + + static final int DEFAULT_METER_MINUTE = 5; + static final int DEFAULT_ASSET_HOUR = 1; + static final int MINUTE = 60 * 1000; + static final int HOUR = 60 * MINUTE; + final int meter; + final int asset; + + public MeteringConfig(int meter, int asset) { + if (meter > 1) this.meter = meter; + else this.meter = DEFAULT_METER_MINUTE; + if (asset > 1) this.asset = asset; + else this.asset = DEFAULT_ASSET_HOUR; + } + + public int getMeterDelay() { + return this.meter * MINUTE; + } + + public int getMeter() { + return this.meter; + } + + public int getAssetDelay() { + return this.asset * HOUR; + } + + public int getAsset() { + return this.asset; + } + + @Override + public String toString() { + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(this); + } catch (JsonProcessingException e) { + return ""; + } + } +} diff --git a/core/backend/src/com/pspace/backend/logManager/metering/SendMetering.java b/core/backend/src/com/pspace/backend/logManager/metering/SendMetering.java new file mode 100644 index 00000000..fbcd9e5e --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/metering/SendMetering.java @@ -0,0 +1,64 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package metering; + +import com.pspace.backend.libs.Utility; + +import db.DBManager; + +public class SendMetering { + + private final MeteringConfig config; + private final DBManager DB; + private boolean Stop = false; + private boolean Quit = false; + + public SendMetering(DBManager DB, MeteringConfig Config) { + this.DB = DB; + this.config = Config; + } + + public void Run() { + int AssetTime = 0; + while (!Quit) { + // 일시정지 확인 + if (Stop) { + Utility.Delay(config.getMeterDelay()); + continue; + } + Utility.Delay(config.getMeterDelay()); + AssetTime += config.getMeterDelay(); + + DB.insertApiMeter(config.getMeter()); + DB.insertIoMeter(config.getMeter()); + + if (AssetTime >= config.getAssetDelay()) { + AssetTime -= config.getAssetDelay(); + + DB.insertApiAsset(); + DB.insertIoAsset(); + } + } + } + + public void Stop() { + Stop = true; + } + + public void Start() { + Stop = false; + } + + public void Quit() { + Quit = true; + } + +} diff --git a/core/backend/src/com/pspace/backend/logManager/pom.xml b/core/backend/src/com/pspace/backend/logManager/pom.xml new file mode 100644 index 00000000..d3fe26df --- /dev/null +++ b/core/backend/src/com/pspace/backend/logManager/pom.xml @@ -0,0 +1,152 @@ + + 4.0.0 + com.pspace.backend.logManager + ksan-log-manager + 1.1.1 + jar + ksanLogManager + + + 1.7.30 + UTF-8 + 11 + + + + + com.google.guava + guava + 30.1-jre + + + org.apache.commons + commons-lang3 + 3.12.0 + + + + com.amazonaws + aws-java-sdk-s3 + 1.11.837 + + + + com.pspace.ifs.ksan.libs + ksan-libs + 0.7.1 + + + com.pspace.ifs.ksan.objmanager + ksan-objmanager + 0.7.1 + + + com.pspace.backend.libs + ksan-backend-libs + 1.1.1 + + + + com.fasterxml.jackson.dataformat + jackson-dataformat-xml + 2.10.3 + + + com.fasterxml.jackson.core + jackson-databind + 2.11.1 + + + + org.ini4j + ini4j + 0.5.4 + + + + ch.qos.logback + logback-classic + 1.2.3 + + + org.slf4j + slf4j-api + ${slf4j.version} + + + org.slf4j + slf4j-simple + ${slf4j.version} + + + + org.elasticsearch + elasticsearch + 7.12.1 + + + org.elasticsearch.client + elasticsearch-rest-high-level-client + 7.12.1 + + + + + ./ + + + maven-compiler-plugin + 3.8.1 + + 11 + + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 3.3.0 + + + ksanLogManager + + + + Main + + + + jar-with-dependencies + + ksanLogManager.jar + + package + + single + + + + + + org.skife.maven + really-executable-jar-maven-plugin + 1.5.0 + + ksanLogManager.jar + + + + package + + really-executable-jar + + + + + + + \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/replication/Filter/ReplicationFilter.java b/core/backend/src/com/pspace/backend/replication/Filter/ReplicationFilter.java new file mode 100644 index 00000000..9c6674c8 --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/Filter/ReplicationFilter.java @@ -0,0 +1,188 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package Filter; + +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Data.Replication.ReplicationEventData; +import com.pspace.backend.libs.Data.S3.S3BucketData; +import com.pspace.backend.libs.Data.S3.S3LogData; +import com.pspace.backend.libs.Data.S3.S3ObjectData; +import com.pspace.backend.libs.Ksan.ObjManagerHelper; +import com.pspace.backend.libs.Ksan.Data.AgentConfig; +import com.pspace.backend.libs.s3format.S3Parameters; +import com.pspace.ifs.ksan.libs.mq.*; + +public class ReplicationFilter implements MQCallback { + private static final Logger logger = LoggerFactory.getLogger(ReplicationFilter.class); + private final AgentConfig ksanConfig; + private final MQSender mq; + + public ReplicationFilter() throws Exception { + this.ksanConfig = AgentConfig.getInstance(); + mq = new MQSender( + ksanConfig.MQHost, + ksanConfig.MQPort, + ksanConfig.MQUser, + ksanConfig.MQPassword, + Constants.MQ_KSAN_LOG_EXCHANGE, + Constants.MQ_EXCHANGE_OPTION_TOPIC, + Constants.MQ_BINDING_REPLICATION_EVENT); + } + + @Override + public MQResponse call(String routingKey, String body) { + + try { + logger.debug("{} -> {}", routingKey, body); + + if (!routingKey.equals(Constants.MQ_BINDING_GW_LOG)) + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + + // 문자열을 Log 클래스로 변환 + var Mapper = new ObjectMapper(); + var s3Log = Mapper.readValue(body, new TypeReference() { + }); + + // 변환 실패할 경우 에러를 전송한다. + if (s3Log == null) { + logger.error("Failed to read S3 Logging"); + return new MQResponse(MQResponseType.ERROR, MQResponseCode.MQ_INVALID_REQUEST, "", 0); + } + + // 버킷 이름이 비어있을 경우 무시 + if (s3Log.isBucketNameEmpty()) + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + + // 기본정보 정의 + var Operation = s3Log.Operation; + var bucketName = s3Log.BucketName; + + // 해당 작업이 에러이거나 복제 대상이 아닐경우 무시 + if (s3Log.isError() || !S3Parameters.ReplicateOperationCheck(Operation)) + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + + // 버킷 정보를 가져온다. + var objManager = ObjManagerHelper.getInstance(); + var bucketInfo = objManager.getBucket(bucketName); + + // Replication 필터링 + if (bucketInfo.isReplication) + ReplicationEventFiltering(bucketInfo, s3Log); + + } catch (Exception e) { + logger.error("", e); + return new MQResponse(MQResponseType.ERROR, MQResponseCode.MQ_UNKNOWN_ERROR, e.getMessage(), 0); + } + + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + } + + private void ReplicationEventFiltering(S3BucketData bucketInfo, S3LogData s3Log) { + try { + // 복제 설정이 없을 경우 스킵 + if (!bucketInfo.isReplication) + return; + + var Operation = s3Log.Operation; // 오퍼레이션 가져오기 + var ReplicationRules = bucketInfo.Replications;// 룰 정보 가져오기 + var sourceBucketName = bucketInfo.BucketName; // 소스버킷 이름 가져오기 + var ObjectName = s3Log.ObjectName; + var versionId = s3Log.VersionId; + + // 룰이 존재하지 않을 경우 스킵 + if (ReplicationRules == null) + return; + + logger.info("Replication Event Check {}", s3Log.Operation); + + // ObjManager 가져오기 + var objManager = ObjManagerHelper.getInstance(); + + // 모든 룰에 대해 필터링 + for (var MyRule : ReplicationRules) { + + try { + if (MyRule == null) { + logger.error("[{}] Replication Rule Invalid!", sourceBucketName); + continue; + } + + // 룰이 활성화 되어있지 않을 경우 스킵 + if (!MyRule.Status) + continue; + + // 오퍼레이션이 Delete이고 설정이 Delete가 활성화 되어있지 않을 경우 스킵 + if (S3Parameters.DeleteOperationCheck(Operation) && !MyRule.DeleteMarker) + continue; + + // 필터 설정이 존재할 경우 + if (MyRule.isFiltering) { + // Prefix 설정이 있다면 + if (StringUtils.isBlank(MyRule.Prefix)) { + // 해당 오브젝트의 Prefix가 일치하지 않을 경우 스킵 + if (!s3Log.ObjectName.startsWith(MyRule.Prefix)) + continue; + + // And 태그 설정이 있다면 + var TagList = MyRule.Tags; + if (TagList.size() > 0) { + // 해당 오브젝트의 태그 정보를 가져오기 + S3ObjectData Object = objManager.getObject(sourceBucketName, ObjectName, versionId); + + // 해당 오브젝트가 존재하지 않을 경우 스킵 + if (Object == null) + continue; + + // 해당 오브젝트의 태그가 존재하지 않을 경우 스킵 + if (Object.tags == null) + continue; + if (Object.tags.size() == 0) + continue; + + int TagCount = 0; + + // 필터에 설정된 모든 태그를 비교 + for (var FilterTag : MyRule.Tags) { + // 필터에 설정된 태그가 오브젝트에 존재하는지 확인 + for (var MyTag : MyRule.Tags) { + // 오브젝트의 태그가 필터에 설정된 태그가 일치할 경우 태그 갯수를 1 증가 + if (MyTag.key == FilterTag.key && MyTag.value == FilterTag.value) + TagCount++; + } + } + + // 해당 오브젝트의 태그목록에 설정한 태그들이 존재하지 않을 경우 스킵 + if (TagCount != MyRule.Tags.size()) + continue; + } + } + } + // 이벤트 저장 + var Data = new ReplicationEventData(Operation, s3Log.ObjectName, s3Log.VersionId, sourceBucketName, + MyRule.TargetBucket, MyRule.TargetRegion); + mq.send(Data.toString(), Constants.MQ_BINDING_REPLICATION_EVENT); + logger.info("Save Event : {}", Data.toString()); + } catch (Exception e) { + logger.error(e.getMessage(), e); + } + } + } catch (Exception e) { + logger.error("", e); + } + } + +} diff --git a/core/backend/src/com/pspace/backend/replication/Main.java b/core/backend/src/com/pspace/backend/replication/Main.java new file mode 100644 index 00000000..ae82cbfd --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/Main.java @@ -0,0 +1,101 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +import java.util.TimeZone; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Utility; +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Heartbeat.Heartbeat; +import com.pspace.backend.libs.Ksan.ObjManagerHelper; +import com.pspace.backend.libs.Ksan.PortalManager; +import com.pspace.backend.libs.Ksan.Data.AgentConfig; + +import Replicator.MainReplicator; +import config.ConfigManager; + +public class Main { + static final Logger logger = LoggerFactory.getLogger(Main.class); + + public static void main(String[] args) { + TimeZone.setDefault(TimeZone.getTimeZone("KST")); + logger.info("Replication Start!"); + + // KSAN 설정을 읽어온다. + var ksanConfig = AgentConfig.getInstance(); + if (!ksanConfig.getConfig()) { + logger.error("Config Read Failed!"); + return; + } + logger.info("ksanAgent Read end"); + + // Get Service Id + var ServiceId = Utility.ReadServiceId(Constants.REPLICATION_SERVICE_ID_PATH); + if (ServiceId == null) { + logger.error("Service Id is Empty"); + return; + } + // Heartbeat + Thread HBThread; + Heartbeat HB; + try { + HB = new Heartbeat(ServiceId, ksanConfig.MQHost, ksanConfig.MQPort, ksanConfig.MQUser, ksanConfig.MQPassword); + HBThread = new Thread(() -> HB.Start(ksanConfig.ServiceMonitorInterval)); + HBThread.start(); + } catch (Exception e) { + logger.error("", e); + return; + } + + // 포탈 초기화 + var portal = PortalManager.getInstance(); + if (!portal.RegionInit()) + { + logger.error("Portal Manager Init Failed!"); + return; + } + logger.info("Portal initialized"); + + // replication 설정을 읽어온다. + var config = ConfigManager.getInstance(); + try { + config.update(); + } catch (Exception e) { + logger.error("", e); + return; + } + logger.info(config.toString()); + + // ObjManager 초기화 + var ObjManager = ObjManagerHelper.getInstance(); + try { + ObjManager.init(config.getObjManagerConfig()); + } catch (Exception e) { + logger.error("", e); + return; + } + + + // Replication Initialization + var Replicator = new MainReplicator(); + if (!Replicator.Start(config.getReplicationUploadThreadCount())) { + logger.error("MainReplicator is not started!"); + return; + } + logger.info("Replicator Start!"); + + while (true) { + Utility.Delay(10000); + } + } +} \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/replication/Replicator/BaseReplicator.java b/core/backend/src/com/pspace/backend/replication/Replicator/BaseReplicator.java new file mode 100644 index 00000000..b377e82f --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/Replicator/BaseReplicator.java @@ -0,0 +1,121 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package Replicator; + +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; + +import com.amazonaws.auth.AWSStaticCredentialsProvider; +import com.amazonaws.auth.BasicAWSCredentials; +import com.amazonaws.client.builder.AwsClientBuilder; +import com.amazonaws.services.s3.AmazonS3; +import com.amazonaws.services.s3.AmazonS3ClientBuilder; +import com.pspace.backend.libs.Utility; +import com.pspace.backend.libs.Data.Replication.ReplicationEventData; +import com.pspace.backend.libs.Ksan.PortalManager; +import com.pspace.backend.libs.Ksan.Data.S3RegionData; +import com.pspace.ifs.ksan.libs.mq.MQCallback; +import com.pspace.backend.libs.Ksan.Data.AgentConfig; + +import config.ConfigManager; + +public abstract class BaseReplicator implements MQCallback { + final Logger logger; + + public enum OperationList { + OP_PUT_OBJECT, OP_PUT_OBJECT_ACL, OP_PUT_OBJECT_RETENTION, OP_PUT_OBJECT_TAGGING, OP_DELETE_OBJECT, + OP_DELETE_OBJECT_TAGGING + } + + protected AmazonS3 SourceClient; + protected AgentConfig ksanConfig; + protected PortalManager portal; + protected ConfigManager config; + + public BaseReplicator(Logger logger) { + this.logger = logger; + this.ksanConfig = AgentConfig.getInstance(); + this.portal = PortalManager.getInstance(); + this.config = ConfigManager.getInstance(); + } + public BaseReplicator(Logger logger, String RegionName) { + this.logger = logger; + this.ksanConfig = AgentConfig.getInstance(); + this.portal = PortalManager.getInstance(); + this.config = ConfigManager.getInstance(); + SetRegion(RegionName); + } + + public boolean SetRegion(String RegionName) { + try { + var SourceRegion = portal.getRegion(RegionName); + if (SourceRegion == null) { + logger.error("Region is not exists"); + return false; + } + + SourceClient = CreateClient(SourceRegion); + if (SourceClient == null) { + logger.error("Source Client is NULL!"); + return false; + } + + return true; + } catch (Exception e) { + logger.error("", e); + return false; + } + } + + /****************************************** + * Utility + *************************************************/ + + protected boolean RegionCheck(String RegionName) { + if (StringUtils.isBlank(RegionName)) + RegionName = config.getRegion(); + + var Region = portal.getRegion(RegionName); + if (Region == null) { + logger.error("Region Name({}) is not exists!", RegionName); + return false; + } + return Utility.S3AliveCheck(Region.getHttpURL()); + } + + protected AmazonS3 CreateClient(S3RegionData Region) { + BasicAWSCredentials awsCreds = new BasicAWSCredentials(Region.AccessKey, Region.SecretKey); + logger.debug("Client : {}, {}", Region.AccessKey, Region.SecretKey); + + return AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(awsCreds)) + .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(Region.getHttpURL(), "")) + .withPathStyleAccessEnabled(true).build(); + } + + protected AmazonS3 CreateClient(ReplicationEventData Data) throws Exception { + return CreateClient(Data.TargetRegion); + } + + protected AmazonS3 CreateClient(String RegionName) throws Exception { + if (StringUtils.isBlank(RegionName)) + return SourceClient; + + var Region = portal.getRegion(RegionName); + if (Region == null) + throw new Exception(RegionName + " Region is not exists."); + + BasicAWSCredentials awsCreds = new BasicAWSCredentials(Region.AccessKey, Region.SecretKey); + + return AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(awsCreds)) + .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(Region.getHttpURL(), "")) + .withPathStyleAccessEnabled(true).build(); + } +} diff --git a/core/backend/src/com/pspace/backend/replication/Replicator/EventReplicator.java b/core/backend/src/com/pspace/backend/replication/Replicator/EventReplicator.java new file mode 100644 index 00000000..9e105332 --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/Replicator/EventReplicator.java @@ -0,0 +1,100 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package Replicator; + +import org.slf4j.LoggerFactory; + +import com.amazonaws.services.s3.AmazonS3; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Data.Replication.ReplicationEventData; +import com.pspace.backend.libs.Data.Replication.ReplicationLogData; +import com.pspace.ifs.ksan.libs.mq.MQResponse; +import com.pspace.ifs.ksan.libs.mq.MQResponseCode; +import com.pspace.ifs.ksan.libs.mq.MQResponseType; +import com.pspace.ifs.ksan.libs.mq.MQSender; + +public class EventReplicator extends BaseReplicator { + + private final MQSender mq; + + public EventReplicator() throws Exception { + super(LoggerFactory.getLogger(EventReplicator.class)); + mq = new MQSender( + ksanConfig.MQHost, + ksanConfig.MQPort, + ksanConfig.MQUser, + ksanConfig.MQPassword, + Constants.MQ_KSAN_LOG_EXCHANGE, + Constants.MQ_EXCHANGE_OPTION_TOPIC, + Constants.MQ_BINDING_REPLICATION_LOG); + } + + public EventReplicator(String RegionName) throws Exception { + super(LoggerFactory.getLogger(EventReplicator.class), RegionName); + mq = new MQSender( + ksanConfig.MQHost, + ksanConfig.MQPort, + ksanConfig.MQUser, + ksanConfig.MQPassword, + Constants.MQ_KSAN_LOG_EXCHANGE, + Constants.MQ_EXCHANGE_OPTION_TOPIC, + Constants.MQ_BINDING_REPLICATION_LOG); + } + + @Override + public MQResponse call(String routingKey, String body) { + try { + logger.debug("{} -> {}", routingKey, body); + + if (!routingKey.equals(Constants.MQ_BINDING_REPLICATION_EVENT)) + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + + // 문자열을 ReplicationEventData 클래스로 변환 + var Mapper = new ObjectMapper(); + var event = Mapper.readValue(body, new TypeReference() { + }); + // 변환 실패시 + if (event == null) { + throw new Exception("Invalid Replication : " + body); + } + + try { + // 목적지 s3의 동작 여부 확인 + if (!RegionCheck(event.TargetRegion)) { + // 동작하지 않을 경우 실패처리 + var data = new ReplicationLogData(event, Constants.EM_S3_NOT_WORKING); + mq.send(data.toString(), Constants.MQ_BINDING_REPLICATION_LOG); + } + + // 타겟 클라이언트 생성 + AmazonS3 TargetClient = CreateClient(event); + + // 전송 객체 생성 + var Sender = new SendReplicator(SourceClient, TargetClient, mq, event, config.getReplicationPartSize()); + + // 복제 시작 + Sender.run(); + } catch (Exception e) { + var data = new ReplicationLogData(event, e.getMessage()); + mq.send(data.toString(), Constants.MQ_BINDING_REPLICATION_LOG); + logger.error("", e); + return new MQResponse(MQResponseType.ERROR, MQResponseCode.MQ_UNKNOWN_ERROR, e.getMessage(), 0); + } + } catch (Exception e) { + logger.error("", e); + return new MQResponse(MQResponseType.ERROR, MQResponseCode.MQ_UNKNOWN_ERROR, e.getMessage(), 0); + } + + return new MQResponse(MQResponseType.SUCCESS, MQResponseCode.MQ_SUCESS, "", 0); + } +} diff --git a/core/backend/src/com/pspace/backend/replication/Replicator/MainReplicator.java b/core/backend/src/com/pspace/backend/replication/Replicator/MainReplicator.java new file mode 100644 index 00000000..4f206505 --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/Replicator/MainReplicator.java @@ -0,0 +1,79 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package Replicator; + +import java.util.ArrayList; +import java.util.List; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Ksan.Data.AgentConfig; +import com.pspace.ifs.ksan.libs.mq.MQReceiver; + +import Filter.ReplicationFilter; +import config.ConfigManager; + +public class MainReplicator { + protected final Logger logger; + protected final ConfigManager config; + protected final AgentConfig agent; + + List filterReceivers = new ArrayList(); + List eventReceivers = new ArrayList(); + + public MainReplicator() { + this.logger = LoggerFactory.getLogger(MainReplicator.class); + this.config = ConfigManager.getInstance(); + this.agent = AgentConfig.getInstance(); + } + + public boolean Start(int ThreadCount) { + try { + for (int index = 0; index < ThreadCount; index++) { + // Filter Receiver 생성 + filterReceivers.add(new MQReceiver( + agent.MQHost, + agent.MQPort, + agent.MQUser, + agent.MQPassword, + Constants.MQ_QUEUE_REPLICATION_S3_LOG, + Constants.MQ_KSAN_LOG_EXCHANGE, + false, + "", + Constants.MQ_BINDING_GW_LOG, + new ReplicationFilter())); + + var eventCallback = new EventReplicator(); + eventCallback.SetRegion(config.getRegion()); + + // Event Receiver 생성 + var eventReceiver = new MQReceiver( + agent.MQHost, + agent.MQPort, + agent.MQUser, + agent.MQPassword, + Constants.MQ_QUEUE_REPLICATION_EVENT_ADD, + Constants.MQ_KSAN_LOG_EXCHANGE, + false, + "", + Constants.MQ_BINDING_REPLICATION_EVENT, + eventCallback); + eventReceivers.add(eventReceiver); + } + return true; + } catch (Exception e) { + logger.error("", e); + return false; + } + } +} diff --git a/core/backend/src/com/pspace/backend/replication/Replicator/SendReplicator.java b/core/backend/src/com/pspace/backend/replication/Replicator/SendReplicator.java new file mode 100644 index 00000000..30637dde --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/Replicator/SendReplicator.java @@ -0,0 +1,394 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package Replicator; + +import java.io.InputStream; +import java.util.ArrayList; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.amazonaws.AmazonServiceException; +import com.amazonaws.services.s3.AmazonS3; +import com.amazonaws.services.s3.model.AccessControlList; +import com.amazonaws.services.s3.model.CanonicalGrantee; +import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest; +import com.amazonaws.services.s3.model.CopyObjectRequest; +import com.amazonaws.services.s3.model.DeleteObjectRequest; +import com.amazonaws.services.s3.model.DeleteObjectTaggingRequest; +import com.amazonaws.services.s3.model.GetBucketAclRequest; +import com.amazonaws.services.s3.model.GetObjectAclRequest; +import com.amazonaws.services.s3.model.GetObjectMetadataRequest; +import com.amazonaws.services.s3.model.GetObjectRequest; +import com.amazonaws.services.s3.model.GetObjectRetentionRequest; +import com.amazonaws.services.s3.model.GetObjectTaggingRequest; +import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; +import com.amazonaws.services.s3.model.ObjectMetadata; +import com.amazonaws.services.s3.model.ObjectTagging; +import com.amazonaws.services.s3.model.PartETag; +import com.amazonaws.services.s3.model.Permission; +import com.amazonaws.services.s3.model.PutObjectRequest; +import com.amazonaws.services.s3.model.S3Object; +import com.amazonaws.services.s3.model.SetObjectAclRequest; +import com.amazonaws.services.s3.model.SetObjectRetentionRequest; +import com.amazonaws.services.s3.model.SetObjectTaggingRequest; +import com.amazonaws.services.s3.model.UploadPartRequest; +import com.pspace.backend.libs.Utility; +import com.pspace.backend.libs.Data.BackendHeaders; +import com.pspace.backend.libs.Data.Constants; +import com.pspace.backend.libs.Data.Replication.ReplicationEventData; +import com.pspace.backend.libs.Data.Replication.ReplicationLogData; +import com.pspace.backend.libs.s3format.S3Parameters; +import com.pspace.ifs.ksan.libs.mq.MQSender; + +public class SendReplicator { + private final Logger logger = LoggerFactory.getLogger(SendReplicator.class); + private final AmazonS3 sourceClient; + private final AmazonS3 targetClient; + private final ReplicationEventData event; + private final MQSender mq; + private final long partSize; + + public SendReplicator(AmazonS3 sourceClient, AmazonS3 targetClient, MQSender mq, ReplicationEventData event, long partSize) { + this.sourceClient = sourceClient; + this.targetClient = targetClient; + this.event = event; + this.partSize = partSize; + this.mq = mq; + } + + public void run() { + try { + // 복제 수행 + String result = Send(); + + // 복제 결과 로그를 저장한다. + var data = new ReplicationLogData(event, result); + mq.send(data.toString(), Constants.MQ_BINDING_REPLICATION_LOG); + logger.info("Save Log : {}", data.toString()); + + } catch (Exception e) { + logger.error("", e); + } + } + + private String Send() { + logger.debug(event.toString()); + if (sourceClient == null) throw new IllegalStateException("Source Client is NULL"); + if (targetClient == null) throw new IllegalStateException("Target Client is NULL"); + + var Result = ""; + var RetryCount = 3; + while (RetryCount > 0) { + try { + switch (event.Operation) { + case S3Parameters.OP_PUT_OBJECT: + PutObject(); + break; + case S3Parameters.OP_PUT_OBJECT_COPY: + CopyObject(); + break; + case S3Parameters.OP_POST_COMPLETE: + MultiPartUpload(); + break; + case S3Parameters.OP_PUT_OBJECT_ACL: + PutObjectACL(); + break; + case S3Parameters.OP_PUT_OBJECT_RETENTION: + PutObjectRetention(); + break; + case S3Parameters.OP_PUT_OBJECT_TAGGING: + PutObjectTagging(); + break; + case S3Parameters.OP_DELETE_OBJECT_TAGGING: + DeleteObjectTagging(); + break; + case S3Parameters.OP_DELETE_OBJECT: + DeleteObject(); + break; + default: + return "Operation Unknown"; + } + return ""; + } catch (AmazonServiceException e) { + Result = String.format("%s(%d) - %s", e.getErrorCode(), e.getStatusCode(), e.getMessage()); + logger.warn("", e); + RetryCount--; + } catch (Exception e) { + Result = e.getMessage(); + logger.warn("", e); + RetryCount--; + } + } + return Result; + } + + /** + * 복제할 대상의 메타 데이터를 설정한다. + * C# API를 사용하여 업로드 할경우 Jetty와의 호환성 이슈로 + * UTF-8이 강제로 대문자 치환되는 버그가 존재하므로 + * 해당 내용에 대응하기 위한 예외처리가 포함되어 있습니다. + * @return Object Metadata + * @throws Exception + */ + protected ObjectMetadata GetObjectMetadata() throws Exception { + var Request = new GetObjectMetadataRequest(event.SourceBucketName, event.ObjectName, event.VersionId); + Request.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + var Metadata = sourceClient.getObjectMetadata(Request); + + // 메타 정보가 비어있을 경우 + if (Metadata == null) { + logger.error("Metadata is Null!"); + throw new Exception("Metadata is Null"); + } + + // UTF-8 sign 에러를 배제하기 위해 대문자로 변경 + if (Metadata.getContentType() != null) + Metadata.setContentType(Metadata.getContentType().replaceAll("UTF-8", "utf-8")); + return Metadata; + } + + /** + * 복제할 대상의 태그 정보를 설정한다. + * @return Object Tagging + * @throws Exception + */ + protected ObjectTagging GetObjectTagging() throws Exception { + var Request = new GetObjectTaggingRequest(event.SourceBucketName, event.ObjectName, event.VersionId); + Request.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + var TagSet = sourceClient.getObjectTagging(Request); + // 태그가 비어있을 경우 + if (TagSet == null) + return null; + return new ObjectTagging(TagSet.getTagSet()); + } + + /** + * 원본 버킷의 권한정보와 타깃버킷의 유저 정보를 가져와서 복제할 대상의 권한 정보를 설정한다. + * @return Object ACL + * @throws Exception + */ + protected AccessControlList GetObjectACL() throws Exception { + // 원본 권한 정보 가져오기 + var SourceRequest = new GetObjectAclRequest(event.SourceBucketName, event.ObjectName, event.VersionId); + SourceRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + var SourceACL = sourceClient.getObjectAcl(SourceRequest); + + // 타겟 버킷 권한 정보 가져오기 + var TargetRequest = new GetBucketAclRequest(event.TargetBucketName); + TargetRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + var TargetACL = targetClient.getBucketAcl(TargetRequest); + var TargetOwner = TargetACL.getOwner(); + + var User = new CanonicalGrantee(TargetOwner.getId()); + User.setDisplayName(TargetOwner.getDisplayName()); + + SourceACL.setOwner(TargetACL.getOwner()); + SourceACL.grantPermission(User, Permission.FullControl); + + return SourceACL; + } + + protected void PutObject() throws Exception { + // 메타 정보 가져오기 + var Metadata = GetObjectMetadata(); + // 태그 정보 받아오기 + var Tagging = GetObjectTagging(); + // 권한 정보 받아오기 + var ACL = GetObjectACL(); + + // 폴더일 경우 오브젝트의 메타데이터만 전송 + InputStream Body = null; + S3Object MyObject = null; + if (FolderCheck(event.ObjectName)) + Body = Utility.CreateBody(""); + // 일반적인 오브젝트일 경우 버전 정보를 포함하여 오브젝트를 다운로드 + else { + var Request = new GetObjectRequest(event.SourceBucketName, event.ObjectName, event.VersionId); + Request.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + MyObject = sourceClient.getObject(Request); + Body = MyObject.getObjectContent(); + } + + // 리퀘스트 생성 + var PutRequest = new PutObjectRequest(event.TargetBucketName, event.ObjectName, Body, Metadata); + // 오브젝트의 태그 정보 등록 + PutRequest.setTagging(Tagging); + // 오브젝트의 ACL 정보 등록 + PutRequest.setAccessControlList(ACL); + + // 오브젝트 Replication + PutRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + PutRequest.putCustomRequestHeader(BackendHeaders.HEADER_REPLICATION, BackendHeaders.HEADER_DATA); + PutRequest.putCustomRequestHeader(BackendHeaders.HEADER_VERSIONID, event.VersionId); + targetClient.putObject(PutRequest); + if (MyObject != null) + MyObject.close(); + } + + protected void CopyObject() throws Exception { + + // 같은 시스템일 경우 복사 + if (sourceClient == targetClient) { + var Request = new CopyObjectRequest(event.SourceBucketName, event.ObjectName, event.TargetBucketName, + event.ObjectName) + .withSourceVersionId(event.VersionId); + Request.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + Request.putCustomRequestHeader(BackendHeaders.HEADER_REPLICATION, BackendHeaders.HEADER_DATA); + + // 복제 + sourceClient.copyObject(Request); + return; + } + throw new IllegalStateException("Anther System CopyObject is not supported."); + + } + + protected void MultiPartUpload() throws Exception { + // 메타 정보 가져오기 + ObjectMetadata Metadata = GetObjectMetadata(); + // 태그 정보 받아오기 + ObjectTagging Tagging = GetObjectTagging(); + // 권한 정보 받아오기 + AccessControlList ACL = GetObjectACL(); + + var InitRequest = new InitiateMultipartUploadRequest(event.TargetBucketName, event.ObjectName, Metadata); + // 오브젝트의 태그 정보 등록 + InitRequest.setTagging(Tagging); + // 오브젝트의 ACL 정보 등록 + InitRequest.setAccessControlList(ACL); + // 헤더추가 + InitRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + // Mulitpart 등록 + var InitResponse = targetClient.initiateMultipartUpload(InitRequest); + var UploadId = InitResponse.getUploadId(); + + // 오브젝트의 사이즈 확인 + var Size = Metadata.getContentLength(); + + // 업로드 시작 + var partList = new ArrayList(); + int PartNumber = 1; + long StartpPosition = 0; + + while (StartpPosition < Size) { + long EndPosition = StartpPosition + partSize; + if (EndPosition > Size) + EndPosition = Size; + + // 업로드할 내용 가져오기 + var Request = new GetObjectRequest(event.SourceBucketName, event.ObjectName) + .withRange(StartpPosition, EndPosition - 1); + Request.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, ""); + var s3Object = sourceClient.getObject(Request); + + // 업로드 파츠 생성 + var PartRequest = new UploadPartRequest() + .withBucketName(event.TargetBucketName) + .withKey(event.ObjectName) + .withUploadId(UploadId) + .withPartNumber(PartNumber++) + .withInputStream(s3Object.getObjectContent()) + .withPartSize(s3Object.getObjectMetadata().getContentLength()); + + // 헤더 추가 + PartRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + PartRequest.putCustomRequestHeader(BackendHeaders.HEADER_REPLICATION, BackendHeaders.HEADER_DATA); + PartRequest.putCustomRequestHeader(BackendHeaders.S3PROXY_HEADER_NO_DR, BackendHeaders.HEADER_DATA); + + var PartResPonse = targetClient.uploadPart(PartRequest); + partList.add(PartResPonse.getPartETag()); + + StartpPosition += partSize; + s3Object.close(); + } + + // 멀티파트 업로드 종료 + var CompRequest = new CompleteMultipartUploadRequest(event.TargetBucketName, event.ObjectName, UploadId, + partList); + + // 헤더추가 + CompRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + CompRequest.putCustomRequestHeader(BackendHeaders.HEADER_REPLICATION, BackendHeaders.HEADER_DATA); + CompRequest.putCustomRequestHeader(BackendHeaders.HEADER_VERSIONID, event.VersionId); + + targetClient.completeMultipartUpload(CompRequest); + } + + protected void PutObjectACL() throws Exception { + // 권한 정보 받아오기 + var ACL = GetObjectACL(); + + // ACL 정보 설정 + var SetRequest = new SetObjectAclRequest(event.TargetBucketName, event.ObjectName, ACL); + // 헤더추가 + SetRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + SetRequest.putCustomRequestHeader(BackendHeaders.HEADER_REPLICATION, BackendHeaders.HEADER_DATA); + targetClient.setObjectAcl(SetRequest); + } + + protected void PutObjectRetention() throws Exception { + // Retention 가져오기 + var GetRequest = new GetObjectRetentionRequest().withBucketName(event.SourceBucketName) + .withKey(event.ObjectName).withVersionId(event.VersionId); + GetRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + var ObjectRetention = sourceClient.getObjectRetention(GetRequest); + + // Retention 설정 + var SetRequest = new SetObjectRetentionRequest().withBucketName(event.TargetBucketName) + .withKey(event.ObjectName).withRetention(ObjectRetention.getRetention()); + // 헤더추가 + SetRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + SetRequest.putCustomRequestHeader(BackendHeaders.HEADER_REPLICATION, BackendHeaders.HEADER_DATA); + targetClient.setObjectRetention(SetRequest); + } + + protected void PutObjectTagging() throws Exception { + // Tagging 가져오기 + var GetRequest = new GetObjectTaggingRequest(event.SourceBucketName, event.ObjectName, + event.VersionId); + GetRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + var Tagging = sourceClient.getObjectTagging(GetRequest); + + // Tagging 설정 + ObjectTagging ObjectTagging = new ObjectTagging(Tagging.getTagSet()); + var SetRequest = new SetObjectTaggingRequest(event.TargetBucketName, event.ObjectName, + ObjectTagging); + // 헤더추가 + SetRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + SetRequest.putCustomRequestHeader(BackendHeaders.HEADER_REPLICATION, BackendHeaders.HEADER_DATA); + targetClient.setObjectTagging(SetRequest); + } + + protected void DeleteObject() throws Exception { + var DelRequest = new DeleteObjectRequest(event.TargetBucketName, event.ObjectName); + // 헤더추가 + DelRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + DelRequest.putCustomRequestHeader(BackendHeaders.HEADER_REPLICATION, BackendHeaders.HEADER_DATA); + targetClient.deleteObject(DelRequest); + } + + protected void DeleteObjectTagging() throws Exception { + var DelRequest = new DeleteObjectTaggingRequest(event.TargetBucketName, event.ObjectName); + // 헤더추가 + DelRequest.putCustomRequestHeader(BackendHeaders.HEADER_BACKEND, BackendHeaders.HEADER_DATA); + DelRequest.putCustomRequestHeader(BackendHeaders.HEADER_REPLICATION, BackendHeaders.HEADER_DATA); + targetClient.deleteObjectTagging(DelRequest); + } + + /****************************************** + * Utility + *************************************************/ + + protected boolean FolderCheck(String ObjectName) { + return ObjectName.endsWith("/"); + } +} diff --git a/core/backend/src/com/pspace/backend/replication/config/ConfigManager.java b/core/backend/src/com/pspace/backend/replication/config/ConfigManager.java new file mode 100644 index 00000000..853cca92 --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/config/ConfigManager.java @@ -0,0 +1,89 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package config; + +import java.io.IOException; + +import com.pspace.backend.libs.Config.ReplicationConfig; +import com.pspace.backend.libs.Ksan.PortalManager; +import com.pspace.ifs.ksan.objmanager.ObjManagerConfig; + +public class ConfigManager { + + private static ReplicationConfig config = null; + private static PortalManager portal = null; + + public static ConfigManager getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static final ConfigManager INSTANCE = new ConfigManager(); + } + + private ConfigManager() { + portal = PortalManager.getInstance(); + } + + public void update() throws IllegalStateException { + config = portal.getReplicationConfig(); + if (config == null) + throw new IllegalStateException("Backend Config is not initialized"); + } + + public ObjManagerConfig getObjManagerConfig() throws IOException { + var item = new ObjManagerConfig(); + item.dbRepository = config.DBType; + item.dbHost = config.DBHost; + item.dbport = config.DBPort; + item.dbName = config.DBName; + item.dbUsername = config.DBUser; + item.dbPassword = config.DBPassword; + + return item; + } + + public String getDBType() { + return config.DBType; + } + + public String getDBHost() { + return config.DBHost; + } + + public int getDBPort() { + return config.DBPort; + } + + public String getDBName() { + return config.DBName; + } + + public String getDBUser() { + return config.DBUser; + } + + public String getDBPassword() { + return config.DBPassword; + } + + public String getRegion() { + return config.Region; + } + + public int getReplicationUploadThreadCount() { + return config.ReplicationUploadThreadCount; + } + + public long getReplicationPartSize() { + return config.ReplicationPartSize; + } +} diff --git a/core/backend/src/com/pspace/backend/replication/docker/DockerFile b/core/backend/src/com/pspace/backend/replication/docker/DockerFile new file mode 100644 index 00000000..231fb603 --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/docker/DockerFile @@ -0,0 +1,27 @@ +FROM maven:3.8.6-jdk-11 as libsbuild +WORKDIR /src +COPY ./core/src/com/pspace/ifs/ksan/libs/ /src/ +RUN --mount=type=cache,target=/root/.m2 mvn -f /src/pom.xml install +######################################################################## +FROM maven:3.8.6-jdk-11 as objmanagerbuild +WORKDIR /src +COPY ./core/src/com/pspace/ifs/ksan/objmanager/ /src/ +RUN --mount=type=cache,target=/root/.m2 mvn -f /src/pom.xml install +######################################################################## +FROM maven:3.8.6-jdk-11 as backendlibsbuild +WORKDIR /src +COPY ./core/backend/src/com/pspace/backend/libs/ /src/ +RUN --mount=type=cache,target=/root/.m2 mvn -f /src/pom.xml install +####################################################################### +FROM maven:3.8.6-jdk-11 as build +WORKDIR /src +COPY ./core/backend/src/com/pspace/backend/replication/ /src/ +RUN --mount=type=cache,target=/root/.m2 mvn -f /src/pom.xml package +######################################################################## +FROM openjdk:11-jre-slim +WORKDIR /app +RUN mkdir -p /usr/local/ksan/etc/ +COPY ./core/backend/src/com/pspace/backend/replication/ksanReplicationManager_log_conf.xml /usr/local/ksan/etc/ +COPY --from=build /src/target/ksanReplicationManager.jar /usr/local/ksan/sbin/ +COPY ./core/backend/src/com/pspace/backend/replication/docker/start.sh /app +CMD ["/app/start.sh"] \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/replication/docker/start.sh b/core/backend/src/com/pspace/backend/replication/docker/start.sh new file mode 100644 index 00000000..9d36b9ea --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/docker/start.sh @@ -0,0 +1,2 @@ +#!/bin/bash +java -jar -Dlogback.configurationFile=/usr/local/ksan/etc/ksanReplicationManager_log_conf.xml /usr/local/ksan/sbin/ksanReplicationManager.jar \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/replication/ksanReplicationManager_log_conf.xml b/core/backend/src/com/pspace/backend/replication/ksanReplicationManager_log_conf.xml new file mode 100644 index 00000000..62e754d8 --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/ksanReplicationManager_log_conf.xml @@ -0,0 +1,24 @@ + + + + + + + %-5level %d{yyyy-MM-dd HH:mm:ss.SSS, ${logback.timezone:-Asia/Seoul}} [%thread{5}][%logger{10}.%method:%line] : %msg%n + + + + ${BACKEND_LOG_PATH}/replication.log + + ${BACKEND_LOG_PATH}/replication.log.%d{yyyy-MM-dd, ${logback.timezone:-Asia/Seoul}}.%i.log.gz + 7 + 100MB + + + %-5level %d{yyyy-MM-dd HH:mm:ss.SSS, ${logback.timezone:-Asia/Seoul}} [%thread{5}][%logger{10}.%method:%line] : %msg%n + + + + + + \ No newline at end of file diff --git a/core/backend/src/com/pspace/backend/replication/pom.xml b/core/backend/src/com/pspace/backend/replication/pom.xml new file mode 100644 index 00000000..d469b538 --- /dev/null +++ b/core/backend/src/com/pspace/backend/replication/pom.xml @@ -0,0 +1,141 @@ + + 4.0.0 + com.pspace.backend.replication + ksan-replication-manager + 1.1.1 + jar + ksanReplicationManager + + + 1.7.30 + UTF-8 + 11 + + + + + com.google.guava + guava + 30.1-jre + + + org.apache.commons + commons-lang3 + 3.12.0 + + + + com.amazonaws + aws-java-sdk-s3 + 1.11.837 + + + + com.pspace.ifs.ksan.libs + ksan-libs + 0.7.1 + + + com.pspace.ifs.ksan.objmanager + ksan-objmanager + 0.7.1 + + + com.pspace.backend.libs + ksan-backend-libs + 1.1.1 + + + + com.fasterxml.jackson.dataformat + jackson-dataformat-xml + 2.10.3 + + + com.fasterxml.jackson.core + jackson-databind + 2.11.1 + + + + org.ini4j + ini4j + 0.5.4 + + + + ch.qos.logback + logback-classic + 1.2.3 + + + org.slf4j + slf4j-api + ${slf4j.version} + + + org.slf4j + slf4j-simple + ${slf4j.version} + + + + + ./ + + + maven-compiler-plugin + 3.8.1 + + 11 + + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 3.3.0 + + + ksanReplicationManager + + + + Main + + + + jar-with-dependencies + + ksanReplicationManager.jar + + package + + single + + + + + + org.skife.maven + really-executable-jar-maven-plugin + 1.5.0 + + ksanReplicationManager.jar + + + + package + + really-executable-jar + + + + + + + \ No newline at end of file diff --git a/core/common/Enums/__init__.py b/core/common/__init__.py similarity index 100% rename from core/common/Enums/__init__.py rename to core/common/__init__.py diff --git a/core/common/common/base_utils.py b/core/common/common/base_utils.py new file mode 100644 index 00000000..2c3a6986 --- /dev/null +++ b/core/common/common/base_utils.py @@ -0,0 +1,347 @@ + +from common.utils import * +from common.define import * +from const.common import * + + +def WaitAgentConfComplete(FuncName, logger, CheckServerId=True, CheckNetworkDevice=False, CheckNetworkId=False): + while True: + ret, conf = GetConf(MonServicedConfPath) + if ret is True: + if conf is not None: + if CheckServerId is True: + if hasattr(conf.__dict__[KeyCommonSection], 'ServerId'): + if not (conf.__dict__[KeyCommonSection].__dict__[KeyServerId] is None or conf.__dict__[KeyCommonSection].__dict__[KeyServerId] == ''): + if CheckNetworkDevice is False: + return conf + else: + if not (conf.__dict__[KeyCommonSection].__dict__[KeyManagementNetDev] is None or conf.__dict__[KeyCommonSection].__dict__[KeyManagementNetDev] == ''): + if CheckNetworkId is False: + return conf + else: + if not (conf.__dict__[KeyCommonSection].__dict__[KeyDefaultNetworkId] is None or conf.__dict__[KeyCommonSection].__dict__[KeyDefaultNetworkId] == ''): + return conf + else: + return conf + + logger.debug('%s wail for %s to be configured. Check = ServerId:True, NetworkDevice:%s CheckNetworkId:%s' % + (FuncName, MonServicedConfPath, str(CheckNetworkDevice), str(CheckNetworkId))) + time.sleep(IntervalLong) + + +def GetAgentConfig(Conf): + if isinstance(Conf, dict): + portalhost = Conf[KeyCommonSection].PortalHost + portalport = Conf[KeyCommonSection].PortalPort + portalapikey = Conf[KeyCommonSection].PortalApiKey + mqhost = Conf[KeyCommonSection].MQHost + mqport = Conf[KeyCommonSection].MQPort + mqpassword = Conf[KeyCommonSection].MQPassword + mquser = Conf[KeyCommonSection].MQUser + serverid = Conf[KeyCommonSection].ServerId + managementnetdev = Conf[KeyCommonSection].ManagementNetDev + defaultnetworkid = Conf[KeyCommonSection].DefaultNetworkId + + serverMonitorInterval = Conf[KeyMonitorSection].ServerMonitorInterval + networkMonitorInterval = Conf[KeyMonitorSection].NetworkMonitorInterval + diskMonitorInterval = Conf[KeyMonitorSection].DiskMonitorInterval + serviceMonitorInterval = Conf[KeyMonitorSection].ServiceMonitorInterval + + + else: + portalhost = Conf.__dict__[KeyCommonSection].__dict__[KeyPortalHost] + portalport = Conf.__dict__[KeyCommonSection].__dict__[KeyPortalPort] + portalapikey = Conf.__dict__[KeyCommonSection].__dict__[KeyPortalApiKey] + mqhost = Conf.__dict__[KeyCommonSection].__dict__[KeyMQHost] + mqport = Conf.__dict__[KeyCommonSection].__dict__[KeyMQPort] + mqpassword = Conf.__dict__[KeyCommonSection].__dict__[KeyMQPassword] + mquser = Conf.__dict__[KeyCommonSection].__dict__[KeyMQUser] + serverid = Conf.__dict__[KeyCommonSection].__dict__[KeyServerId] + managementnetdev = Conf.__dict__[KeyCommonSection].__dict__[KeyManagementNetDev] + defaultnetworkid = Conf.__dict__[KeyCommonSection].__dict__[KeyDefaultNetworkId] + + serverMonitorInterval = Conf.__dict__[KeyMonitorSection].__dict__[KeyServerMonitorInterval] + networkMonitorInterval = Conf.__dict__[KeyMonitorSection].__dict__[KeyNetworkMonitorInterval] + diskMonitorInterval = Conf.__dict__[KeyMonitorSection].__dict__[KeyDiskMonitorInterval] + serviceMonitorInterval = Conf.__dict__[KeyMonitorSection].__dict__[KeyServiceMonitorInterval] + + # Key conversion(ksanAgen.conf -> Conf object attribute + class Config: pass + setattr(Config, 'PortalHost', portalhost) + setattr(Config, 'PortalPort' , portalport ) + setattr(Config, 'PortalApiKey', portalapikey) + setattr(Config, 'MQHost', mqhost) + setattr(Config, 'MQPort', mqport) + setattr(Config, 'MQPassword', mqpassword) + setattr(Config, 'MQUser', mquser) + setattr(Config, 'ServerId', serverid) + setattr(Config, 'ManagementNetDev', managementnetdev) + setattr(Config, 'DefaultNetworkId', defaultnetworkid) + + setattr(Config, 'ServerMonitorInterval', serverMonitorInterval) + setattr(Config, 'NetworkMonitorInterval', networkMonitorInterval) + setattr(Config, 'DiskMonitorInterval', diskMonitorInterval) + setattr(Config, 'ServiceMonitorInterval', serviceMonitorInterval) + + return Config + + + + +def Byte2HumanValue(Byte, Title, Color=True): + Byte = float(Byte) + if Title == 'TotalSize': + if (int(Byte) / OneTBUnit) > 0.99: # TB + TB = int(Byte) / OneTBUnit + if Color is True: + return (clr.okbl + "%8.1f" % round(TB, 1) + 'T' + clr.end) + else: + return "%8.1f" % round(TB, 1) + 'T' + elif (int(Byte) / OneGBUnit) > 0.99: # GB + GB = int(Byte) / OneGBUnit + if Color is True: + return (clr.okbl + "%8.1f" % round(GB, 1) + 'G' + clr.end) + else: + return "%8.1f" % round(GB, 1) + 'G' + elif (int(Byte) / OneMBUnit) > 0.99: # MB + MB = int(Byte) / OneMBUnit + if Color is True: + return (clr.okgr + "%8.1f" % round(MB, 1)+'M' + clr.end) + else: + return "%8.1f" % round(MB, 1) + 'M' + + elif (int(Byte) / OneKBUnit) > 0.99: # MB + MB = int(Byte) / OneKBUnit + if Color is True: + return (clr.warnye + "%8.1f" % round(MB, 1) + 'K' + clr.end) + else: + return "%8.1f" % round(MB, 1) + 'K' + else:# KB + KB = int(Byte) + if Color is True: + return (clr.badre + "%8.1f" % round(KB, 1)+'B' + clr.end) + else: + return "%8.1f" % round(KB, 1) + 'B' + elif Title == 'UsedSize' or Title == 'FreeSize': + if (int(Byte) / OneTBUnit) > 0.99: # TB + TB = int(Byte) / OneTBUnit + if Color is True: + return (clr.okbl + "%7.1f" % round(TB, 1) + 'T' + clr.end) + else: + return "%7.1f" % round(TB, 1) + 'T' + + elif (int(Byte) / OneGBUnit) > 0.99: # GB + GB = int(Byte) / OneGBUnit + if Color is True: + return (clr.okbl + "%7.1f" % round(GB, 1) + 'G' + clr.end) + else: + return "%7.1f" % round(GB, 1) + 'G' + elif (int(Byte) / OneMBUnit) > 0.99: # MB + MB = int(Byte) / OneMBUnit + if Color is True: + return (clr.okgr + "%7.1f" % round(MB, 1) + 'M' + clr.end) + else: + return "%7.1f" % round(MB, 1) + 'M' + + elif (int(Byte) / OneKBUnit) > 0.99: # MB + MB = int(Byte) / OneKBUnit + if Color is True: + return (clr.warnye + "%7.1f" % round(MB, 1) + 'K' + clr.end) + else: + return "%7.1f" % round(MB, 1) + 'K' + else: # KB + KB = int(Byte) + if Color is True: + return (clr.badre + "%7.1f" % round(KB, 1) + 'B' + clr.end) + else: + return "%7.1f" % round(KB, 1) + 'B' + elif Title == 'DiskRw': + if (int(Byte) / OneTBUnit) > 0.99: # TB + TB = int(Byte) / OneTBUnit + if Color is True: + return (clr.okbl + "%7.1f" % round(TB, 1) + 'T' + clr.end) + else: + return "%7.1f" % round(TB, 1) + 'T' + elif (int(Byte) / OneGBUnit) > 0.99: # GB + GB = int(Byte) / OneGBUnit + if Color is True: + return (clr.okbl + "%7.1f" % round(GB, 1) + 'G' + clr.end) + else: + return "%7.1f" % round(GB, 1) + 'G' + elif (int(Byte) / OneMBUnit) > 0.99: # MB + MB = int(Byte) / OneMBUnit + if Color is True: + return (clr.okgr + "%7.1f" % round(MB, 1) + 'M' + clr.end) + else: + return "%7.1f" % round(MB, 1) + 'M' + + elif (int(Byte) / OneKBUnit) > 0.99: # MB + MB = int(Byte) / OneKBUnit + if Color is True: + return (clr.warnye + "%7.1f" % round(MB, 1) + 'K' + clr.end) + else: + return "%7.1f" % round(MB, 1) + 'K' + else: # KB + KB = int(Byte) + if Color is True: + return (clr.badre + "%7.1f" % round(KB, 1) + 'B' + clr.end) + else: + return "%7.1f" % round(KB, 1) + 'B' + + +def DisplayDiskState(State): + if State == DiskStart: + return (clr.okgr + "%-7s" % State + clr.end) + elif State == DiskStop: + return (clr.badre + "%-7s" % State + clr.end) + elif State == DiskWeak: + return (clr.warnye + "%-7s" % State + clr.end) + elif State == DiskDisable: + return (clr.badre + "%-7s" % State + clr.end) + +def DisplayState(State, WhiteSpace=10): + if State in [ServiceStateOnline, ServerStateOnline]: + return (clr.okgr + "%-10s" % State + clr.end) + elif State in [ServerStateOffline, ServerStateTimeout, ServiceStateOffline, ServiceStateUnkown]: + return (clr.badre + "%-10s" % State + clr.end) + + + +def DisplayDiskMode(Mode): + if Mode == DiskModeRw: + return DiskModeRwShort + else: + return DiskModeRoShort + + + +@catch_exceptions() +class GetApiResult: + def __init__(self, Header, ItemHeader, Items): + self.Header = Header + self.ItemHeader = ItemHeader + self.Items = Items + + +class ResponseHeader(object): + """ + Parsing Response without "Data" value is single value like True/False + """ + def __init__(self): + self.IsNeedLogin = None + self.AccessDenied = None + self.Result = None + self.Code = None + self.Message = None + + def Set(self, IsNeedLogin, AccessDenied, Result, Code, Message): + self.IsNeedLogin = IsNeedLogin + self.AccessDenied = AccessDenied + self.Result = Result + self.Code = Code + self.Message = Message + +class ResponseHeaderWithData(object): + """ + Parsing Response with "Data" value is multi value like dict or list + """ + def __init__(self): + self.IsNeedLogin = None + self.AccessDenied = None + self.Result = None + self.Code = None + self.Message = None + self.Data = None + + def Set(self, IsNeedLogin, AccessDenied, Result, Code, Message, Data): + self.IsNeedLogin = IsNeedLogin + self.AccessDenied = AccessDenied + self.Result = Result + self.Code = Code + self.Message = Message + self.Data = Data + + +class ResponseItemsHeader(object): + def __init__(self): + self.TotalCount = '' + self.Skips = '' + self.PageNo = '' + self.CountPerPage = '' + self.PagePerSection = '' + self.TotalPage = '' + self.StartPageNo = '' + self.EndPageNo = '' + self.PageNos = '' + self.HavePreviousPage = '' + self.HaveNextPage = '' + self.HavePreviousPageSection = '' + self.HaveNextPageSection = '' + self.Items = '' + + +@catch_exceptions() +class ResPonseHeader(object): + def __init__(self, dic, logger=None): + try: + self.IsNeedLogin = dic['IsNeedLogin'] + self.AccessDenied = dic['AccessDenied'] + self.Result = dic['Result'] + self.Code = dic['Code'] + self.Message = dic['Message'] + if 'Data' in dic: + self.Data = dic['Data'] + except KeyError as err: + if logger is not None: + logger.error(err, sys.exc_info()[2].tb_lineno) + else: + print(err, sys.exc_info()[0].tb_lineno) + + +##### Response Body Decoder Class ##### +class ResPonseHeaderDecoder(json.JSONDecoder): + def __init__(self, logger=None, *args, **kwargs): + # json.JSONDecoder.__init__(self, object_hook=self.object_hook, *args, **kwargs) + self.logger=logger + json.JSONDecoder.__init__(self, object_hook=self.object_hook, *args, **kwargs) + + def object_hook(self, dct): + try: + print(">>>>", dct) + if 'IsNeedLogin' in dct: + obj = ResPonseHeader(dct) + if 'Data' in dct: + print(dct['Data']) + return obj + except KeyError as err: + if self.logger is not None: + self.logger.error(err, sys.exc_info()[2].tb_lineno) + else: + print(err, sys.exc_info()[0].tb_lineno) + + +##### Items Header of Response Body Class ##### +class ResPonseItemsHeader(object): + def __init__(self, dic, logger=None): + try: + self.TotalCount = dic['TotalCount'] + self.Skips = dic['Skips'] + self.PageNo = dic['PageNo'] + self.CountPerPage = dic['CountPerPage'] + self.PagePerSection = dic['PagePerSection'] + self.TotalPage = dic['TotalPage'] + self.StartPageNo = dic['StartPageNo'] + self.EndPageNo = dic['EndPageNo'] + self.PageNos = dic['PageNos'] + self.HavePreviousPage = dic['HavePreviousPage'] + self.HaveNextPage = dic['HaveNextPage'] + self.HavePreviousPageSection = dic['HavePreviousPageSection'] + self.HaveNextPageSection = dic['HaveNextPageSection'] + self.Items = dic['Items'] + except KeyError as err: + if logger is not None: + logger.error(err, sys.exc_info()[2].tb_lineno) + else: + print(err, sys.exc_info()[0].tb_lineno) + + diff --git a/core/common/common/daemon.py b/core/common/common/daemon.py deleted file mode 100644 index 262ce6c6..00000000 --- a/core/common/common/daemon.py +++ /dev/null @@ -1,101 +0,0 @@ -#!/usr/bin/pyhon3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import os, sys -import atexit -import signal -import time -from common.utils import IsDaemonRunning - -class Daemon(object): - def __init__(self, PidFile, DaemonName): - self.PidFile = PidFile - self.DaemonName = DaemonName - self.redirect = '/dev/null' - self.pid = -1 - - def daemonize(self): - try: - pid = os.fork() - if pid > 0: - # exit first parent - sys.exit(0) - except OSError as e: - sys.stderr.write("fork failed: %d (%s)\n" % (e.errno, e.strerror)) - sys.exit(1) - - # decouple from parent environment - os.chdir("/") - os.setsid() - os.umask(0) - - # do second fork - try: - pid = os.fork() - if pid > 0: - # exit from second parent - sys.exit(0) - except OSError as e: - self.pid = pid - sys.stderr.write("fork failed: %d (%s)\n" % (e.errno, e.strerror)) - sys.exit(1) - - # write pidfile - atexit.register(self.delpid) - self.pid = str(os.getpid()) - if self.PidFile: - with open(self.PidFile, 'w+') as f: - f.write("%s" % self.pid) - - # redirect standard file descriptors - sys.stdout.flush() - sys.stderr.flush() - sio = open(self.redirect, 'a+') - os.dup2(sio.fileno(), sys.stdin.fileno()) - os.dup2(sio.fileno(), sys.stdout.fileno()) - os.dup2(sio.fileno(), sys.stderr.fileno()) - - sio.close() - - def delpid(self): - if self.PidFile and os.path.exists(self.PidFile): - os.remove(self.PidFile) - - def start(self): - # Start the daemon - self.daemonize() - self.run() - - def stop(self): - Ret, Pid = IsDaemonRunning(self.PidFile, self.DaemonName) - if Ret is False: - return - try: - while 1: - os.kill(int(Pid), signal.SIGKILL) - time.sleep(0.1) - except OSError as err: - err = str(err) - if err.find("No such process") > 0: - if os.path.exists(self.PidFile): - os.remove(self.PidFile) - else: - sys.exit(1) - - def restart(self): - self.stop() - self.start() - - def run(self): - sys.stderr.write("run:\n") - pass diff --git a/core/common/common/define.py b/core/common/common/define.py index e69de29b..bf2ed63d 100644 --- a/core/common/common/define.py +++ b/core/common/common/define.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +""" +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +""" + + +##### FILE FORMAT TYPE Define ##### +ConfigTypeINI = 'INI' +ConfigTypeObject = 'Object' + + + +##### Path Define ##### +ProcDiskStatsPath = '/proc/diskstats' + + +""" +##### HTTP ##### +""" +### http header key ### +HeaderContentType = 'Content-Type' +HeaderAuth = 'Authorization' + + +### http method ### +GetMethod = 'GET' +PostMethod = 'POST' +PutMethod = 'PUT' +DeleteMethod = 'DELETE' + + + + +##### ENCODING ##### +UTF8 = 'utf-8' + + + + +##### Header of Response Body Class for deserialization ##### +ResponseItemsHeaderModule = 'common.base_utils.ResponseItemsHeader' +ResponseHeaderModule = 'common.base_utils.ResponseHeader' +ResponseHeaderWithDataModule = 'common.base_utils.ResponseHeaderWithData' + diff --git a/core/common/common/dictcasttest.py b/core/common/common/dictcasttest.py deleted file mode 100644 index 747f17eb..00000000 --- a/core/common/common/dictcasttest.py +++ /dev/null @@ -1,91 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - - -import jsonpickle -import json - -data = { - "Server": { - "Id": "1dec24c0-6e14-4a12-a0ca-c62c68377f56", - "Name": "Dev Server 3", - }, - "Id": "a10f5043-5973-477d-9556-f0ec2f60f75b", - "ServerId": "1dec24c0-6e14-4a12-a0ca-c62c68377f56", - "DiskPoolId": '', - "DiskNo": "1366638474", -} - -data1 = {"Id":"1", "Name":"2", "py/object": "__main__.Data1"} - -class Server: - def __init__(self, Id, Name): - self.Id = Id - self.Name = Name - -class Disk: - def __init__(self): - self.Server = Server - self.Id = '' - self.DiskPoolId = '' - self.DiskNo = '' - self.a = '' - -class Data1: - def __init__(self, Id, Name): - self.Id = Id - self.Name = Name -#aa= jsonpickle.encode(data, make_refs=True) -#print(aa) - -data.update({"py/object":"__main__.Disk"}) - -aa = jsonpickle.decode(json.dumps(data)) -print(aa) -import pdb -pdb.set_trace() - -class SubObject: - def __init__(self, sub_name, sub_age): - self.sub_name = sub_name - self.sub_age = sub_age - - -class TestClass: - - def __init__(self, name, age, sub_object): - self.name = name - self.age = age - self.sub_object = sub_object - - -john_junior = SubObject("John jr.", 2) - -john = TestClass("John", 21, john_junior) - -file_name = 'JohnWithSon' + '.json' - -john_string = jsonpickle.encode(john) - -with open(file_name, 'w') as fp: - fp.write(john_string) - -john_from_file = open(file_name).read() - -test_class_2 = jsonpickle.decode(john_from_file) - -print(test_class_2.name) -print(test_class_2.age) -print(test_class_2.sub_object.sub_name) - - diff --git a/core/common/common/display.py b/core/common/common/display.py deleted file mode 100644 index 12be203a..00000000 --- a/core/common/common/display.py +++ /dev/null @@ -1,127 +0,0 @@ -#!/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -from common.define import * -from common.log import catch_exceptions - - -@catch_exceptions() -def disp_serverinfo(data, Id=None): - - title =""" - %s%s%s%s%s%s%s%s%s - """ % ('Name'.center(20), 'Description'.center(30), 'CpuModel'.center(30), 'Clock'.center(20), 'State'.center(20), 'ModeDate'.center(20), 'ModId'.center(20), 'ModName'.center(20), 'Id'.center(30)) - print(title) - if Id is None: - for svrinfo in data.Items: - svr = ServerItems() - svr.Set(svrinfo) - _svr ="%s%s%s%s%s%s%s%s%s" % \ - (svr.Name.center(20), '{:20.20}'.format(svr.Description.center(30)), '{:20.20}'.format(svr.CpuModel.center(30)), str(svr.Clock).center(20), svr.State.center(20), svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(_svr) - else: - _svr = "%-20s%20s%20s%5s%10s%15s%10s%10s%20s" % \ - (data.Name, data.Description, data.CpuModel, data.Clock, data.State, data.ModDate, data.ModId, data.ModName, data.Id) - print(_svr) - -@catch_exceptions() -def disp_network_interfaces(data, NicId=None): - title =""" - %s%s%s%s%s%s%s - """ % ('Id'.center(40), 'ServerId'.center(40), 'Nic'.center(20), 'Description'.center(30), 'IpAddress'.center(20), 'MacAddress'.center(20), 'LinkStatus'.center(20)) # 'ModeDate'.center(20), 'ModId'.center(20), 'ModName'.center(20), 'Id'.center(30)) - print(title) - if NicId is None: - for nicinfo in data.Items: - nic = ResponseNetworkInterfaceItems(nicinfo) - _nic ="%s%s%s%s%s%s%s" % (nic.Id.center(40), '{:40.40}'.format(nic.ServerId.center(40)), - '{:20.20}'.format(nic.Name.center(20)),'{:30.30}'.format(nic.Description.center(30)), '{:20.20}'.format(nic.IpAddress.center(20)), nic.MacAddress.center(20), str(nic.LinkState).center(20)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(_nic) - else: - - _nic = "%s%s%s%s%s%s%s" % (data.Id.center(40), '{:40.40}'.format(data.ServerId.center(40)), - '{:20.20}'.format(data.Name.center(20)), - '{:30.30}'.format(data.Description.center(30)), - '{:20.20}'.format(data.IpAddress.center(20)), data.MacAddress.center(20), - data.LinkState.center( - 20)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(_nic) - - -@catch_exceptions() -def disp_vlan_network_interfaces(data, NicId=None): - title =""" - %s%s%s%s%s%s%s - """ % ('Id'.center(40), 'InterfaceId'.center(40), 'Tag'.center(20), 'IpAddress'.center(20), 'SubnetMask'.center(20), 'Gateway'.center(20), 'RegDate'.center(20)) # 'ModeDate'.center(20), 'ModId'.center(20), 'ModName'.center(20), 'Id'.center(30)) - print(title) - if NicId is None: - for vlaninfo in data.Items: - vlan = ResPonseVlanNetworkInterfaceItems(vlaninfo) - _vlan ="%s%s%s%s%s%s%s" % (vlan.Id.center(40), '{:40.40}'.format(vlan.InterfaceId.center(40)), - '{:20.20}'.format(str(vlan.Tag).center(20)),'{:30.30}'.format(vlan.IpAddress.center(20)), '{:20.20}'.format(vlan.SubnetMask.center(20)), vlan.Gateway.center(20), str(vlan.RegDate).center(20)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(_vlan) - else: - - _vlan= "%s%s%s%s%s%s%s" % (data.Id.center(40), '{:40.40}'.format(data.InterfaceId.center(40)), - '{:20.20}'.format(str(data.Tag).center(20)), - '{:30.30}'.format(data.IpAddress.center(20)), - '{:20.20}'.format(data.SubnetMask.center(20)), data.Gateway.center(20), - str(data.RegDate).center( - 20)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(_vlan) - - -@catch_exceptions() -def disp_disk_info(data, DiskId=None): - """ - display disk info with server - :param data: if DiskId is None, DiskInfoPerServer object list. otherwise DiskItems object. - :param DiskId: - :return: - """ - title =""" - %s%s%s%s%s%s%s%s%s%s%s - """ % ('ServerId'.center(40), 'Id'.center(40), 'DiskNo'.center(20), 'Path'.center(20), 'HaAction'.center(20), 'State'.center(20), 'DiskType'.center(20), 'Total'.center(15), 'Used'.center(15), 'Free'.center(15), 'RwMode'.center(10)) # 'ModeDate'.center(20), 'ModId'.center(20), 'ModName'.center(20), 'Id'.center(30)) - print(title) - if DiskId is None: - """ - for diskinfo in data.Items: - disk = DiskItems() - disk.Set(diskinfo) - _disk ="%s%s%s%s%s%s%s%s%s%s%s" % (disk.ServerId.center(40), '{:40.40}'.format(disk.Id.center(40)), - '{:20.20}'.format(str(disk.DiskNo).center(20)), - '{:30.30}'.format(disk.Path.center(20)), '{:20.20}'.format(disk.HaAction.center(20)), - disk.State.center(20), str(disk.DiskType).center(20), str(disk.TotalSize).center(15), - str(disk.UsedSize).center(15),str(disk.ReservedSize).center(15), - disk.RwMode.center(10) ) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(_disk) - """ - for svr in data: - for disk in svr.List: - - _dsp ="%s%s%s%s%s%s%s%s%s%s%s" % (svr.Name.center(40), '{:40.40}'.format(disk.Id.center(40)), - '{:20.20}'.format(str(disk.DiskNo).center(20)), - '{:30.30}'.format(disk.Path.center(20)), '{:20.20}'.format(disk.HaAction.center(20)), - disk.State.center(20), str(disk.DiskType).center(20), str(disk.TotalSize).center(15), - str(disk.UsedSize).center(15),str(disk.ReservedSize).center(15), - disk.RwMode.center(10)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(_dsp) - else: - - _disk = "%s%s%s%s%s%s%s%s%s%s%s" % (data.ServerId.center(40), '{:40.40}'.format(data.Id.center(40)), - '{:20.20}'.format(str(data.DiskNo).center(20)), - '{:30.30}'.format(data.Path.center(20)), - '{:20.20}'.format(data.HaAction.center(20)), - data.State.center(20), str(data.DiskType).center(20), - str(data.TotalSize).center(15), str(data.UsedSize).center(15), - str(data.ReservedSize).center(15), data.RwMode.center(10)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(_disk) - diff --git a/core/common/common/httpapi.py b/core/common/common/httpapi.py deleted file mode 100644 index 98de39be..00000000 --- a/core/common/common/httpapi.py +++ /dev/null @@ -1,351 +0,0 @@ -#!/usr/bin/env python3 -# -*- coding: utf-8 -*- -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import sys -import os -import requests -import urllib3 -if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: - sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from const.common import * -from const.http import ResponseHeaderWithDataModule, ResponseItemsHeaderModule, ResponseHeaderModule, ResPonseHeader, \ - ResPonseItemsHeader, GetApiResult, Parsing -from common.log import Logging, catch_exceptions -import json -import jsonpickle -import pdb - -#AuthKey = "5de46d7ccd5d0954fad7d11ffc22a417e2784cbedd9f1dae3992a46e97b367e8" -IfsPortalIp = None -IfsPortalPort = None - -# http response status -CONTINUE = 100 -SWITCHING_PROTOCOLS = 101 -PROCESSING = 102 -OK = 200 -CREATED = 201 -ACCEPTED = 202 -NON_AUTHORITATIVE_INFORMATION = 203 -NO_CONTENT = 204 -RESET_CONTENT = 205 -PARTIAL_CONTENT = 206 -MULTI_STATUS = 207 -IM_USED = 226 -MULTIPLE_CHOICES = 300 -MOVED_PERMANENTLY = 301 -FOUND = 302 -SEE_OTHER = 303 -NOT_MODIFIED = 304 -USE_PROXY = 305 -TEMPORARY_REDIRECT = 307 -BAD_REQUEST = 400 -UNAUTHORIZED = 401 -PAYMENT_REQUIRED = 402 -FORBIDDEN = 403 -NOT_FOUND = 404 -METHOD_NOT_ALLOWED = 405 -NOT_ACCEPTABLE = 406 -PROXY_AUTHENTICATION_REQUIRED = 407 -REQUEST_TIMEOUT = 408 -CONFLICT = 409 -GONE = 410 -LENGTH_REQUIRED = 411 -PRECONDITION_FAILED = 412 -REQUEST_ENTITY_TOO_LARGE = 413 -REQUEST_URI_TOO_LONG = 414 -UNSUPPORTED_MEDIA_TYPE = 415 -REQUESTED_RANGE_NOT_SATISFIABLE = 416 -EXPECTATION_FAILED = 417 -UNPROCESSABLE_ENTITY = 422 -LOCKED = 423 -FAILED_DEPENDENCY = 424 -UPGRADE_REQUIRED = 426 -PRECONDITION_REQUIRED = 428 -TOO_MANY_REQUESTS = 429 -REQUEST_HEADER_FIELDS_TOO_LARGE = 431 -INTERNAL_SERVER_ERROR = 500 -NOT_IMPLEMENTED = 501 -BAD_GATEWAY = 502 -SERVICE_UNAVAILABLE = 503 -GATEWAY_TIMEOUT = 504 -HTTP_VERSION_NOT_SUPPORTED = 505 -INSUFFICIENT_STORAGE = 507 -NOT_EXTENDED = 510 -NETWORK_AUTHENTICATION_REQUIRED = 511 - - -class RestApi: - - def __init__(self, ip, port, url, header=None, params=None, protocol='https', authkey=None, logger=None): - self._ip = ip - self._port = port - self._params = params - self._protocol = protocol - self._url = url - self.logger = logger - if self.logger is None: - self.logger = Logging().get_logger() - - if header is None: - header = dict() - header[HeaderAuth] = authkey - - self._header = header - if params is None: - params = dict() - self._params = params - - def get(self, ItemsHeader=True, ReturnType=None): - url = "%s://%s:%d%s" % (self._protocol, self._ip, self._port, self._url) - urllib3.disable_warnings() - err_msg = '' - try: - r = requests.get(url, params=self._params, verify=False, headers=self._header) - self.logging_request_info(GetMethod, r) - ret = r.content.decode(UTF8) - Data = json.loads(ret) - Ret = self.GetResponse(Data,ItemsHeader=ItemsHeader, ReturnType=ReturnType) - return ResOk, '', Ret - except requests.ConnectionError as err: - self.logger.error("Connection Error %s" % err) - return ResConnectionErrorCode, ResConnectionErrorMsg,None - except requests.URLRequired as err: - self.logger.error("UrlRequired Error %s" % err) - return ResInvalidCode, ResInvalidMsg, None - except (requests.ConnectTimeout, requests.ReadTimeout, requests.Timeout) as err: - self.logger.error("Timeout Error %s" % err) - return ResTimeErrorCode, ResTimeErrorCodeMsg, None - except Exception as err: - self.logger.error("Other Error %s" % err) - return ResEtcErrorCode, ResEtcErrorMsg, None - - def post(self,ItemsHeader=True, ReturnType=None): - url = "%s://%s:%d%s" % (self._protocol, self._ip, self._port, self._url) - urllib3.disable_warnings() - try: - self._header[HeaderContentType] = 'application/json' - # self._params = json.dumps(self._params) - r = requests.post(url=url, data=self._params, headers=self._header, verify=False) - self.logging_request_info(PostMethod, r) - ret = r.content.decode(UTF8) - Data = json.loads(ret) - Ret = self.GetResponse(Data,ItemsHeader=ItemsHeader, ReturnType=ReturnType) - return ResOk, '', Ret - - except requests.ConnectionError as err: - self.logger.error("Connection Error %s" % err) - return ResConnectionErrorCode, ResTimeErrorCodeMsg,None - except requests.URLRequired as err: - self.logger.error("UrlRequired Error %s" % err) - return ResInvalidCode, ResInvalidMsg, None - except (requests.ConnectTimeout, requests.ReadTimeout, requests.Timeout) as err: - self.logger.error("Timeout Error %s" % err) - return ResTimeErrorCode, ResTimeErrorCodeMsg, None - except Exception as err: - self.logger.error("Other Error %s" % err) - return ResEtcErrorCode, ResEtcErrorMsg, None - - def put(self,ItemsHeader=True, ReturnType=None): - url = "%s://%s:%d%s" % (self._protocol, self._ip, self._port, self._url) - urllib3.disable_warnings() - try: - self._header[HeaderContentType] = 'application/json-patch+json' - #self._params = json.dumps(self._params) - r = requests.put(url=url, data=self._params, headers=self._header, verify=False) - self.logging_request_info(PutMethod, r) - ret = r.content.decode(UTF8) - Data = json.loads(ret) - Ret = self.GetResponse(Data,ItemsHeader=ItemsHeader, ReturnType=ReturnType) - return ResOk, '', Ret - except requests.ConnectionError as err: - self.logger.error("Connection Error %s" % err) - return ResConnectionErrorCode, ResTimeErrorCodeMsg,None - except requests.URLRequired as err: - self.logger.error("UrlRequired Error %s" % err) - return ResInvalidCode, ResInvalidMsg, None - except (requests.ConnectTimeout, requests.ReadTimeout, requests.Timeout) as err: - self.logger.error("Timeout Error %s" % err) - return ResTimeErrorCode, ResTimeErrorCodeMsg, None - except Exception as err: - self.logger.error("Other Error %s" % err) - return ResEtcErrorCode, ResEtcErrorMsg, None - - @catch_exceptions() - def delete(self, ItemsHeader=True, ReturnType=None): - url = "%s://%s:%d%s" % (self._protocol, self._ip, self._port, self._url) - urllib3.disable_warnings() - try: - self._header[HeaderContentType] = 'application/json-patch+json' - #self._params = json.dumps(self._params) - r = requests.delete(url=url, data=self._params, headers=self._header, verify=False) - self.logging_request_info(DeleteMethod, r) - ret = r.content.decode(UTF8) - Data = json.loads(ret) - Ret = self.GetResponse(Data,ItemsHeader=ItemsHeader, ReturnType=ReturnType) - return ResOk, '', Ret - - except requests.ConnectionError as err: - self.logger.error("Connection Error %s" % err) - return ResConnectionErrorCode, ResTimeErrorCodeMsg,None - except requests.URLRequired as err: - self.logger.error("UrlRequired Error %s" % err) - return ResInvalidCode, ResInvalidMsg, None - except (requests.ConnectTimeout, requests.ReadTimeout, requests.Timeout) as err: - self.logger.error("Timeout Error %s" % err) - return ResTimeErrorCode, ResTimeErrorCodeMsg, None - except Exception as err: - self.logger.error("Other Error %s" % err) - return ResEtcErrorCode, ResEtcErrorMsg, None - @catch_exceptions() - def logging_request_info(self, method, req): - try: - self.logger.debug("Request %s url >>>>> %s" % (method, req.url)) - self.logger.debug("Header : %s" % req.request.headers) - self.logger.debug("Body : %s" % req.request.body) - except Exception as err: - print(err) - - def logging_response_info(self, Data): - try: - self.logger.debug("Response Header <<<<< %s" % str(Data)) - except Exception as err: - print(err) - - @catch_exceptions() - def GetResponse(self, Ret, ItemsHeader=False, ReturnType=None): - """ - Convert Dict Result to ResponseHeader Class - There are three kinds of ResponseHeader - - single type1(data:True/False) - - sigle type2() - - multi value(dict)) - """ - self.logging_response_info(Ret) - if 'Data' in Ret: - Ret.update({"py/object": ResponseHeaderWithDataModule}) - if Ret['Data'] is None: # if data items not exists. - Ret = jsonpickle.decode(json.dumps(Ret)) - return Ret - - if ItemsHeader is True: # if items header exists - Ret['Data'].update({"py/object": ResponseItemsHeaderModule}) - for Item in Ret['Data']['Items']: - DeserializeResponseReculsive(Item, ReturnType) - else: - DeserializeResponseReculsive(Ret['Data'], ReturnType) - - else: - # Return - DeserializeResponseReculsive(Ret, ResponseHeaderModule) - - Ret = jsonpickle.decode(json.dumps(Ret)) - return Ret - - def GetResponse1(self, Data, ItemsHeader=False): - self.logging_response_info(Data) - Data.update({"py/object": ResponseHeaderModule}) - if ItemsHeader is True: - Data['Data'].update({"py/object": ResponseItemsHeaderModule}) - - Ret = jsonpickle.decode(json.dumps(Data)) - return Ret - - @catch_exceptions() - def parsing_result(self, header): - ''' - cast total result dict to class(header, item header, data) - :param header: dict - :return: GetApiResult Object - ''' - Header = None - ItemsHeader = None - Data = None - try: - self.logger.debug("Response << %s" % str(header)) - Header = ResPonseHeader(header) - if "Data" in Header.__dict__: - if isinstance(Header.Data, dict): - if "Items" in Header.Data: - ItemsHeader = ResPonseItemsHeader(Header.Data) - Data = DictToObject(Header.Data) - else: - Data = Header.Data - # print header attr - self.logger.debug("Header: %s" % str(Header.__dict__)) - - # print ItemsHeader attr - if hasattr(ItemsHeader, '__dict__'): - dict_attr = str(ItemsHeader.__dict__) - else: - dict_attr = 'None' - self.logger.debug("ItemsHeader: %s" % dict_attr) - - # print Data attr - - if hasattr(Data, '__dict__'): - dict_attr = str(Data.__dict__) - else: - dict_attr = 'None' - self.logger.debug("Data: %s" % dict_attr) - - except Exception as err: - self.logger.error("fail to parsing response data: %s line:%d" % (str(err), sys.exc_info()[2].tb_lineno)) - finally: - return GetApiResult(Header, ItemsHeader, Data) - - -@catch_exceptions() -def DeserializeResponse(Data, ObjectType): - """ - Get Data parsing with ObjectType Class - :param Data: Dict type, - :param ObjectType: Class name - :return: ObjectType class - """ - Data.update({"py/object": ObjectType}) - Ret = jsonpickle.decode(json.dumps(Data)) - return Ret - - -def DeserializeResponseReculsive(Data, ObjectType): - """ - Get Data parsing with ObjectType Class - :param Data: Dict type, - :param ObjectType: Class name - :return: ObjectType class - """ - if ObjectType is not None: - Data.update({"py/object": ObjectType}) - for key, val in Data.items(): - if key in Parsing.keys(): - if isinstance(val, list): - for val1 in Data[key]: - DeserializeResponseReculsive(val1, Parsing[key]) - else: - if isinstance(val, dict): - Data[key].update({"py/object": Parsing[key]}) - - -def get_res(res): - - print("IsNeedLogin:", res.IsNeedLogin) - print("AccessDenied:", res.AccessDenied) - print("Result:", res.Result) - print("Code:", res.Code) - print("Message:", res.Message) - if 'Data' in res.__dict__: - if not isinstance(res.Data, dict): - print("Data:", res.Data) - diff --git a/core/common/common/init.py b/core/common/common/init.py deleted file mode 100644 index c7358f6b..00000000 --- a/core/common/common/init.py +++ /dev/null @@ -1,304 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import os, sys -import fcntl -import re -import getpass -import time -import pdb -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from const.common import * -from configparser import ConfigParser -from common.shcommand import GetHostInfo -from common.log import * -from common.utils import IsIpValid, IsUrlValid - - -def get_input(Description, type, default=None, ValidAnsList=None, force=True): - """ - get info from user - :param Description: - :param type: - :param default: - :param ValidAnsList: - :return: - """ - try: - while True: - if type == 'pwd': - ans = getpass.getpass(Description + '(default: %s):' % str(default) if default is not None else ':') - else: - ans = input(Description + '(default: %s):' % str(default) if default is not None else ':') - - if ans in ['', None]: - if type in ['host']: - ans = default - - if ans: - if type == int: - if ans.isdigit(): - return int(ans) - else: - print('Invalid Digit type') - elif type == 'ip': - if IsIpValid(ans): - return ans - else: - print('Invalid Ip Type') - elif type == 'host': - if IsIpValid(ans): - ret, hostname, ip = GetHostInfo(ip=ans) - else: - ret, hostname, ip = GetHostInfo(hostname=ans) - - if ret is False: - print('Invalid Host') - continue - else: - return ip - - elif type == 'url': - if IsUrlValid(ans): - return ans - else: - print('Invalid Url Type') - else: - if ValidAnsList is not None: - if ans not in ValidAnsList: - print('Invalid Answer Type. Only %s is supported' % ', '.join(ValidAnsList)) - continue - return ans - else: - if default == '': - if type in ['pwd', 'net_device', 'host']: - print('Invalid %s type' % type) - continue - - if default is not None: - return default - except Exception as err: - print(err, sys.exc_info()[2].tb_lineno) - - -@catch_exceptions() -def read_conf(path): - """ - read conf file - """ - if not os.path.exists(path): - print('init first') - return False, None - - config = ConfigParser(delimiters="=", strict=True, allow_no_value=True) - config.optionxform = str - config.read(path) - conf = dict() - for section in config.sections(): - conf[section] = dict() - for option in config.options(section): - conf[section][option] = config.get(section, option) - return True, conf - - -@catch_exceptions() -def read_conf_normal(path, fo=None, lock_sh=False): - """ - normal key=value type conf file - :param path: - :param fo: - :param lock_sh: - :return: dict - """ - - config = {} - openfile = True - if fo: openfile = False - if openfile: - try: - fo = open(path) - except IOError as err: - return False, config - if lock_sh: fcntl.flock(fo.fileno(), fcntl.LOCK_SH) - lines = fo.readlines() - for line in lines: - #mo = re.search('(\w+)\s*=\s*((?:\w|/|\.\-_)+)', line) - #if mo is None and not line.isspace(): - # #mo = re.search('(\w*)\s*=\s*\"((?:\w|\s|-|:|\(|\)\._\-)+)\"', line) - mo = re.search("([\d\w_\-\.]+)[\s]{0,2}=[\s]{0,2}([/\d\w_\-\.:]+)", line) - if mo: - config[mo.group(1)] = mo.group(2) - if lock_sh: fcntl.flock(fo.fileno(), fcntl.LOCK_UN) - if openfile: fo.close() - return True, config - - -@catch_exceptions() -def GetConf(filename, FileType=ConfigTypeINI, ReturnType=ConfigTypeObject): - """ - filename: configuration file Path - """ - if not os.path.exists(filename): - return False, None - if FileType == ConfigTypeINI: - Res, Conf = read_conf(filename) - else: - Res, Conf = read_conf_normal(filename) - - if Res is True: - if ReturnType == ConfigTypeObject: - return Res, DictToObject(Conf) - else: - # return dictionary type - return Res, Conf - - else: - return Res, None - -def isValidConfig(ConfPath, *CheckKeys, **kwargs): - """ - check if conf is valid with CheckKey - :param ConfPath: file path - :param **kwargs: check if the value is valid with CheckKeys - :return: bool - """ - - ret, conf = GetConf(ConfPath) - if ret is False: - return ret, conf - else: - isValid = True - for key in CheckKeys: - if conf.mgs.__dict__[key] is None or conf.mgs.__dict__[key] == '': - isValid = False - - for key, val in kwargs: - if conf.mgs.__dict__[key] != val: - isValid = False - - return isValid, conf - - - -def UpdateConf(Path, Section, Key, Value, logger, Force=True): - """ - Update NetowrkId and ServerId in ksanMonitor.conf - :param Path: - :param Section: - :param Key: - :param Value: - :param Force: - :return: - """ - res, conf = read_conf(Path) - if res is True: - if Section not in conf: - logger.error('Invalid Section: %s in %s' % (Section, Path)) - return False - if Key not in conf[Section] and Force is False: - logger.error('Invalid Key: %s in %s' % (Key, Path)) - return False - conf[Section][Key] = Value - config = ConfigParser() - config.optionxform = str - for section in conf.keys(): - config[section] = dict() - for key, val in conf[section].items(): - config[section][key] = val - - with open(Path, 'w') as configfile: - config.write(configfile) - return True - else: - return False - -def WaitAgentConfComplete(FuncName, logger, CheckServerId=True, CheckNetworkDevice=False, CheckNetworkId=False): - while True: - ret, conf = GetConf(MonServicedConfPath) - if ret is True: - if conf is not None: - if CheckServerId is True: - if hasattr(conf.__dict__[KeyCommonSection], 'ServerId'): - if not (conf.__dict__[KeyCommonSection].__dict__[KeyServerId] is None or conf.__dict__[KeyCommonSection].__dict__[KeyServerId] == ''): - if CheckNetworkDevice is False: - return conf - else: - if not (conf.__dict__[KeyCommonSection].__dict__[KeyManagementNetDev] is None or conf.__dict__[KeyCommonSection].__dict__[KeyManagementNetDev] == ''): - if CheckNetworkId is False: - return conf - else: - if not (conf.__dict__[KeyCommonSection].__dict__[KeyDefaultNetworkId] is None or conf.__dict__[KeyCommonSection].__dict__[KeyDefaultNetworkId] == ''): - return conf - else: - return conf - - logger.debug('%s wail for %s to be configured. Check = ServerId:True, NetworkDevice:%s CheckNetworkId:%s' % - (FuncName, MonServicedConfPath, str(CheckNetworkDevice), str(CheckNetworkId))) - time.sleep(IntervalLong) - - -def GetAgentConfig(Conf): - if isinstance(Conf, dict): - portalhost = Conf[KeyCommonSection].PortalHost - portalport = Conf[KeyCommonSection].PortalPort - portalapikey = Conf[KeyCommonSection].PortalApiKey - mqhost = Conf[KeyCommonSection].MQHost - mqport = Conf[KeyCommonSection].MQPort - mqpassword = Conf[KeyCommonSection].MQPassword - mquser = Conf[KeyCommonSection].MQUser - serverid = Conf[KeyCommonSection].ServerId - managementnetdev = Conf[KeyCommonSection].ManagementNetDev - defaultnetworkid = Conf[KeyCommonSection].DefaultNetworkId - - serverMonitorInterval = Conf[KeyMonitorSection].ServerMonitorInterval - networkMonitorInterval = Conf[KeyMonitorSection].NetworkMonitorInterval - diskMonitorInterval = Conf[KeyMonitorSection].DiskMonitorInterval - serviceMonitorInterval = Conf[KeyMonitorSection].ServiceMonitorInterval - - - else: - portalhost = Conf.__dict__[KeyCommonSection].__dict__[KeyPortalHost] - portalport = Conf.__dict__[KeyCommonSection].__dict__[KeyPortalPort] - portalapikey = Conf.__dict__[KeyCommonSection].__dict__[KeyPortalApiKey] - mqhost = Conf.__dict__[KeyCommonSection].__dict__[KeyMQHost] - mqport = Conf.__dict__[KeyCommonSection].__dict__[KeyMQPort] - mqpassword = Conf.__dict__[KeyCommonSection].__dict__[KeyMQPassword] - mquser = Conf.__dict__[KeyCommonSection].__dict__[KeyMQUser] - serverid = Conf.__dict__[KeyCommonSection].__dict__[KeyServerId] - managementnetdev = Conf.__dict__[KeyCommonSection].__dict__[KeyManagementNetDev] - defaultnetworkid = Conf.__dict__[KeyCommonSection].__dict__[KeyDefaultNetworkId] - - serverMonitorInterval = Conf.__dict__[KeyMonitorSection].__dict__[KeyServerMonitorInterval] - networkMonitorInterval = Conf.__dict__[KeyMonitorSection].__dict__[KeyNetworkMonitorInterval] - diskMonitorInterval = Conf.__dict__[KeyMonitorSection].__dict__[KeyDiskMonitorInterval] - serviceMonitorInterval = Conf.__dict__[KeyMonitorSection].__dict__[KeyServiceMonitorInterval] - - # Key conversion(ksanAgen.conf -> Conf object attribute - class Config: pass - setattr(Config, 'PortalHost', portalhost) - setattr(Config, 'PortalPort' , portalport ) - setattr(Config, 'PortalApiKey', portalapikey) - setattr(Config, 'MQHost', mqhost) - setattr(Config, 'MQPort', mqport) - setattr(Config, 'MQPassword', mqpassword) - setattr(Config, 'MQUser', mquser) - setattr(Config, 'ServerId', serverid) - setattr(Config, 'ManagementNetDev', managementnetdev) - setattr(Config, 'DefaultNetworkId', defaultnetworkid) - - setattr(Config, 'ServerMonitorInterval', serverMonitorInterval) - setattr(Config, 'NetworkMonitorInterval', networkMonitorInterval) - setattr(Config, 'DiskMonitorInterval', diskMonitorInterval) - setattr(Config, 'ServiceMonitorInterval', serviceMonitorInterval) - - return Config \ No newline at end of file diff --git a/core/common/common/log.py b/core/common/common/log.py deleted file mode 100644 index 74adaea9..00000000 --- a/core/common/common/log.py +++ /dev/null @@ -1,66 +0,0 @@ -#!/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import sys, os, traceback -import os, sys -if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: - sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -import logging -from functools import wraps -import pdb - -LogFilePath = '/var/log/ksan/agent/agent.log' - -class Logging(object): - def __init__(self, instance=__name__, logfile=LogFilePath, loglevel='error'): - try: - self.logfile = logfile - if loglevel == 'debug': - self.loglevel = logging.DEBUG - else: - self.loglevel = logging.ERROR - self.logger = logging.getLogger(instance) - self.set_loglevel() - formatter = logging.Formatter('%(asctime)s %(filename)s(%(funcName)s:%(lineno)d) %(levelname)s - %(message)s') - self.fh = logging.FileHandler(logfile) - self.fh.setFormatter(formatter) - self.logger.addHandler(self.fh) - except Exception as err: - print(err, sys.exc_info()[2].tb_lineno) - - def set_loglevel(self): - self.logger.setLevel(self.loglevel) - logging.addLevelName(45, 'INFO') - logging.INFO = 45 - - def create(self): - return self.logger - - def get_logger(self, name=__name__): - return logging.getLogger(name) - - -def catch_exceptions(): - logger = logging.getLogger('common.log') - - def wrapper(func): - @wraps(func) - def decorateor(*args, **kwargs): - try: - result = func(*args, **kwargs) - return result - except Exception as err: - exc_type, exc_obj, exc_tb = sys.exc_info() - logger.error("%s %s %s " % (str(err), traceback.extract_tb(exc_tb)[-1][0], traceback.extract_tb(exc_tb)[-1][1])) - return None - return decorateor - return wrapper diff --git a/core/common/common/network.py b/core/common/common/network.py deleted file mode 100644 index 5d09aa82..00000000 --- a/core/common/common/network.py +++ /dev/null @@ -1,108 +0,0 @@ -#!/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os -import sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -import time -import datetime -import psutil -import netifaces -import dns.resolver -from common.httpapi import * - - -class GetNetwork(object): - def __init__(self, Ip=None, logger=None): - self.logger = logger - self.NicInfo = None - self.nicinfo_list = list() - self.GetNicInfo(Ip=Ip) - self.IoCounterPerNic = psutil.net_io_counters(pernic=True) - - @catch_exceptions() - def GetNicInfo(self, Ip=None): - """ - get all detail nic info list. - :return: - """ - # get dns info - dnsresolvers = dns.resolver.Resolver() - - # get status of each nic(isup(tru/false), duplex, speed, mtu) - netif_stat = psutil.net_if_stats() - # get nic info. netif_addresses includes {'eno1':..., 'eno2':...} - netif_addresses = psutil.net_if_addrs() - for nic in netif_addresses.keys(): - if nic == 'lo' or netifaces.AF_INET not in netifaces.ifaddresses(nic): - continue - tmp_dic = dict() - tmp_dic['Name'] = nic - tmp_dic['Dhcp'] = 'No' # must 'No' - tmp_dic['MacAddress'] = netifaces.ifaddresses(nic)[netifaces.AF_LINK][0]['addr'] - tmp_dic['LinkState'] = 'Up' if netif_stat[nic].isup is True else 'Down' - tmp_dic['IpAddress'] = netifaces.ifaddresses(nic)[netifaces.AF_INET][0]['addr'] - tmp_dic['SubnetMask'] = netif_addresses[nic][0].netmask - tmp_dic['BandWidth'] = netif_stat[nic].speed - tmp_dic['Gateway'] = self.GetGatewayWithNic(nic) - tmp_dic['Dns1'] = '' - tmp_dic['Dns2'] = '' - for idx, nameserver in enumerate(dnsresolvers.nameservers, start=1): - key = 'Dns%d' % idx - tmp_dic[key] = nameserver - - if tmp_dic['Dns2'] == '': - tmp_dic['Dns2'] = tmp_dic['Dns1'] - - - self.nicinfo_list.append(tmp_dic) - if Ip is not None: - if Ip == tmp_dic['IpAddress']: - self.NicInfo = tmp_dic - return tmp_dic - - @catch_exceptions() - def GetGatewayWithNic(self, NicName): - gw = netifaces.gateways() - for gateway, nic, is_default in gw[netifaces.AF_INET]: - if NicName == nic: - return gateway - return '' - - - @catch_exceptions() - def GetNicInfoWithNicName(self, NicName): - """ - return specific nic info with nic name - :return: success to get nic info {'name', , 'Dhcp': , 'MacAddress':, 'LinkState': , - 'addr',
, 'netask': , 'gateway': , 'Dns1': , 'Dns2': , 'Dns3':...}, fail to get nic info: None - """ - if NicName is None: - print('nic is None') - else: - for nic in self.nicinfo_list: - if nic['Name'] == NicName: - return nic - return None - - - def ReNewalNicStat(self): - """ - Update Nic stat - :return: - """ - self.IoCounterPerNic = psutil.net_io_counters(pernic=True) - self.GetNicInfo() - - - - diff --git a/core/common/common/shcommand.py b/core/common/common/shcommand.py deleted file mode 100644 index b5955069..00000000 --- a/core/common/common/shcommand.py +++ /dev/null @@ -1,116 +0,0 @@ -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import pdb -from subprocess import Popen, PIPE -import socket -import re - -def shcall(command): - """ - Executes a command and returns the result - """ - try: - p = Popen(command, shell=True, stdout=PIPE, stderr=PIPE, close_fds=True, universal_newlines=True) - except (OSError, ValueError) as err: - return '', err - return p.communicate() - - -def GetHostInfo(hostname=None, ip=None): - try: - if hostname is None: - hostname = socket.gethostname() - if ip is None: - ip = socket.gethostbyname(hostname) - return True, hostname, ip - except socket.error as err: - return False, str(err), None - -def isStringIp(String): - IpFinder = re.compile("([\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3})") - IpType = IpFinder.search(String) - if IpType: - return True - else: - return False - - -def GetIpFromHostname(String): - """ - if String is ip type, return String otherwise return ip from String - :param hostname: - :return: - """ - if isStringIp(String): - return String - else: - return GetHostInfo(hostname=String) - - -def UpdateEtcHosts(ClusterMembers:list, Mode): # ClusterMembers: [('192.168.11.11', 'osd1')] - OmittedHostNameList = list() - lines = '' - with open("/etc/hosts", 'r') as f: - lines = f.readlines() - - if Mode == 'add': - for member in ClusterMembers: - ip, hostname = member - IpFinder = re.compile("^%s[\s]" % ip) - HostNameFinder1 = re.compile("[\s]%s[\s]" % hostname) - HostNameFinder2 = re.compile("[\s]%s$" % hostname) - Found = False - for line in lines: - trimedlist = line.rsplit() - trimedstring = " ".join(trimedlist) - ValidIp = IpFinder.search(trimedstring) - HostNameType1 = HostNameFinder1.search(trimedstring) - HostNameType2 = HostNameFinder2.search(trimedstring) - if ValidIp and (HostNameType1 or HostNameType2): - Found = True - break - if Found is False: - OmittedHostNameList.append("%s %s" % (ip, hostname)) - if len(OmittedHostNameList): - with open("/etc/hosts", 'a+') as f: - for hostname in OmittedHostNameList: - f.write("%s\n" % hostname) - - elif Mode == 'remove': - UpdatedLines = '' - ip, hostname = ClusterMembers[0] - HostNameFinder1 = re.compile("[\s]%s[\s]" % hostname) - HostNameFinder2 = re.compile("[\s]%s$" % hostname) - - Removed = False - for line in lines: - if line == '\n' or line == ' ': - continue - - trimedlist = line.rsplit() - trimedstring = " ".join(trimedlist) - - HostNameType1 = HostNameFinder1.search(trimedstring) - HostNameType2 = HostNameFinder2.search(trimedstring) - if HostNameType1 or HostNameType2: - Removed = True - continue - - else: - UpdatedLines += line - - if Removed is True: - with open("/etc/hosts", 'w') as f: - f.write("%s\n" % UpdatedLines) - else: - print('Invalid Mode') diff --git a/core/common/common/utils.py b/core/common/common/utils.py index 433dd632..85234f38 100644 --- a/core/common/common/utils.py +++ b/core/common/common/utils.py @@ -12,11 +12,72 @@ import os, sys import re -from common.shcommand import shcall -from const.common import OneTBUnit, OneGBUnit, OneMBUnit, OneKBUnit, DiskStart, DiskStop, DiskWeak, DiskDisable, \ - DiskModeRo, DiskModeRw, DiskModeRoShort, DiskModeRwShort, ServerStateOffline, ServerStateOnline, \ - ServiceStateOffline, ServiceStateOnline, ServiceStateUnkown, ServerStateTimeout +from subprocess import Popen, PIPE +import socket import signal +import logging +from functools import wraps +import traceback +from common.define import * +from const.common import * +from configparser import ConfigParser +import fcntl +import getpass +import requests +import urllib3 +import json, jsonpickle +import netifaces +import dns.resolver +import atexit + + + +LogFilePath = '/var/log/ksan/agent/agent.log' + +class Logging(object): + def __init__(self, instance=__name__, logfile=LogFilePath, loglevel='error'): + try: + self.logfile = logfile + if loglevel == 'debug': + self.loglevel = logging.DEBUG + else: + self.loglevel = logging.ERROR + self.logger = logging.getLogger(instance) + self.set_loglevel() + formatter = logging.Formatter('%(asctime)s %(filename)s(%(funcName)s:%(lineno)d) %(levelname)s - %(message)s') + self.fh = logging.FileHandler(logfile) + self.fh.setFormatter(formatter) + self.logger.addHandler(self.fh) + except Exception as err: + print(err, sys.exc_info()[2].tb_lineno) + + def set_loglevel(self): + self.logger.setLevel(self.loglevel) + logging.addLevelName(45, 'INFO') + logging.INFO = 45 + + def create(self): + return self.logger + + def get_logger(self, name=__name__): + return logging.getLogger(name) + + +def catch_exceptions(): + logger = logging.getLogger('common.log') + + def wrapper(func): + @wraps(func) + def decorateor(*args, **kwargs): + try: + result = func(*args, **kwargs) + return result + except Exception as err: + exc_type, exc_obj, exc_tb = sys.exc_info() + logger.error("%s %s %s " % (str(err), traceback.extract_tb(exc_tb)[-1][0], traceback.extract_tb(exc_tb)[-1][1])) + return None + return decorateor + return wrapper def AvoidSigInt(): @@ -27,6 +88,109 @@ def SigIntHandler(signum, frame): sys.exit(-1) + +def shcall(command): + """ + Executes a command and returns the result + """ + try: + p = Popen(command, shell=True, stdout=PIPE, stderr=PIPE, close_fds=True, universal_newlines=True) + except (OSError, ValueError) as err: + return '', err + return p.communicate() + + +def GetHostInfo(hostname=None, ip=None): + try: + if hostname is None: + hostname = socket.gethostname() + if ip is None: + ip = socket.gethostbyname(hostname) + return True, hostname, ip + except socket.error as err: + return False, str(err), None + +def isStringIp(String): + IpFinder = re.compile("([\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3})") + IpType = IpFinder.search(String) + if IpType: + return True + else: + return False + + +def GetIpFromHostname(String): + """ + if String is ip type, return String otherwise return ip from String + :param hostname: + :return: + """ + if isStringIp(String): + return String + else: + return GetHostInfo(hostname=String) + + +def UpdateEtcHosts(ClusterMembers:list, Mode): # ClusterMembers: [('192.168.11.11', 'osd1')] + OmittedHostNameList = list() + lines = '' + with open("/etc/hosts", 'r') as f: + lines = f.readlines() + + if Mode == 'add': + for member in ClusterMembers: + ip, hostname = member + IpFinder = re.compile("^%s[\s]" % ip) + HostNameFinder1 = re.compile("[\s]%s[\s]" % hostname) + HostNameFinder2 = re.compile("[\s]%s$" % hostname) + Found = False + for line in lines: + trimedlist = line.rsplit() + trimedstring = " ".join(trimedlist) + ValidIp = IpFinder.search(trimedstring) + HostNameType1 = HostNameFinder1.search(trimedstring) + HostNameType2 = HostNameFinder2.search(trimedstring) + if ValidIp and (HostNameType1 or HostNameType2): + Found = True + break + if Found is False: + OmittedHostNameList.append("%s %s" % (ip, hostname)) + if len(OmittedHostNameList): + with open("/etc/hosts", 'a+') as f: + for hostname in OmittedHostNameList: + f.write("%s\n" % hostname) + + elif Mode == 'remove': + UpdatedLines = '' + ip, hostname = ClusterMembers[0] + HostNameFinder1 = re.compile("[\s]%s[\s]" % hostname) + HostNameFinder2 = re.compile("[\s]%s$" % hostname) + + Removed = False + for line in lines: + if line == '\n' or line == ' ': + continue + + trimedlist = line.rsplit() + trimedstring = " ".join(trimedlist) + + HostNameType1 = HostNameFinder1.search(trimedstring) + HostNameType2 = HostNameFinder2.search(trimedstring) + if HostNameType1 or HostNameType2: + Removed = True + continue + + else: + UpdatedLines += line + + if Removed is True: + with open("/etc/hosts", 'w') as f: + f.write("%s\n" % UpdatedLines) + else: + print('Invalid Mode') + + + def IsDaemonRunning(PidPath, CmdLine=None): """ Check if pid exists in /proc and cmdline is valid. @@ -214,131 +378,779 @@ class clr: end = '\033[0m' -def Byte2HumanValue(Byte, Title, Color=True): - if Title == 'TotalSize': - if (int(Byte) / OneTBUnit) > 0.99: # TB - TB = int(Byte) / OneTBUnit - if Color is True: - return (clr.okbl + "%8.1f" % round(TB, 1) + 'T' + clr.end) - else: - return "%8.1f" % round(TB, 1) + 'T' - elif (int(Byte) / OneGBUnit) > 0.99: # GB - GB = int(Byte) / OneGBUnit - if Color is True: - return (clr.okbl + "%8.1f" % round(GB, 1) + 'G' + clr.end) - else: - return "%8.1f" % round(GB, 1) + 'G' - elif (int(Byte) / OneMBUnit) > 0.99: # MB - MB = int(Byte) / OneMBUnit - if Color is True: - return (clr.okgr + "%8.1f" % round(MB, 1)+'M' + clr.end) - else: - return "%8.1f" % round(MB, 1) + 'M' +##### Conversion Dict to Object Class ##### +@catch_exceptions() +class DictToObject(object): - elif (int(Byte) / OneKBUnit) > 0.99: # MB - MB = int(Byte) / OneKBUnit - if Color is True: - return (clr.warnye + "%8.1f" % round(MB, 1) + 'K' + clr.end) + def __init__(self, myDict): + for key, value in myDict.items(): + if type(value) == dict: + setattr(self, key, DictToObject(value)) else: - return "%8.1f" % round(MB, 1) + 'K' - else:# KB - KB = int(Byte) - if Color is True: - return (clr.badre + "%8.1f" % round(KB, 1)+'B' + clr.end) - else: - return "%8.1f" % round(KB, 1) + 'B' - elif Title == 'UsedSize' or Title == 'FreeSize': - if (int(Byte) / OneTBUnit) > 0.99: # TB - TB = int(Byte) / OneTBUnit - if Color is True: - return (clr.okbl + "%7.1f" % round(TB, 1) + 'T' + clr.end) - else: - return "%7.1f" % round(TB, 1) + 'T' + if isinstance(value, str) and value.isdigit(): + value = int(value) + setattr(self, key, value) - elif (int(Byte) / OneGBUnit) > 0.99: # GB - GB = int(Byte) / OneGBUnit - if Color is True: - return (clr.okbl + "%7.1f" % round(GB, 1) + 'G' + clr.end) - else: - return "%7.1f" % round(GB, 1) + 'G' - elif (int(Byte) / OneMBUnit) > 0.99: # MB - MB = int(Byte) / OneMBUnit - if Color is True: - return (clr.okgr + "%7.1f" % round(MB, 1) + 'M' + clr.end) + +@catch_exceptions() +def read_conf(path): + """ + read conf file + """ + if not os.path.exists(path): + print('init first') + return False, None + + config = ConfigParser(delimiters="=", strict=True, allow_no_value=True) + config.optionxform = str + config.read(path) + conf = dict() + for section in config.sections(): + conf[section] = dict() + for option in config.options(section): + conf[section][option] = config.get(section, option) + return True, conf + + +@catch_exceptions() +def read_conf_normal(path, fo=None, lock_sh=False): + """ + normal key=value type conf file + :param path: + :param fo: + :param lock_sh: + :return: dict + """ + + config = {} + openfile = True + if fo: openfile = False + if openfile: + try: + fo = open(path) + except IOError as err: + return False, config + if lock_sh: fcntl.flock(fo.fileno(), fcntl.LOCK_SH) + lines = fo.readlines() + for line in lines: + #mo = re.search('(\w+)\s*=\s*((?:\w|/|\.\-_)+)', line) + #if mo is None and not line.isspace(): + # #mo = re.search('(\w*)\s*=\s*\"((?:\w|\s|-|:|\(|\)\._\-)+)\"', line) + if line.endswith("=\n") or line.endswith("= \n"): + mo = re.search("([\d\w_\-\.]+)[\s]{0,2}=[\s]{0,10}", line) + if mo: + config[mo.group(1)] = '' + else: + mo = re.search("([\d\w_\-\.]+)[\s]{0,2}=[\s]{0,2}([/\d\w_\-\.:]+)", line) + if mo: + config[mo.group(1)] = mo.group(2) + + if lock_sh: fcntl.flock(fo.fileno(), fcntl.LOCK_UN) + if openfile: fo.close() + return True, config + + + +@catch_exceptions() +def GetConf(filename, FileType=ConfigTypeINI, ReturnType=ConfigTypeObject): + """ + filename: configuration file Path + """ + if not os.path.exists(filename): + return False, None + if FileType == ConfigTypeINI: + Res, Conf = read_conf(filename) + else: + Res, Conf = read_conf_normal(filename) + + if Res is True: + if ReturnType == ConfigTypeObject: + return Res, DictToObject(Conf) + else: + # return dictionary type + return Res, Conf + + else: + return Res, None + + + +def get_input(Description, type, default=None, ValidAnsList=None, ValueComment='', force=True): + """ + get info from user + :param Description: + :param type: + :param default: + :param ValidAnsList: + :return: + """ + try: + while True: + if type == 'pwd': + ans = getpass.getpass(Description + '(%sdefault: %s):' % (ValueComment, str(default) if default is not None else ':')) else: - return "%7.1f" % round(MB, 1) + 'M' + ans = input(Description + '(%sdefault: %s):' % (ValueComment, str(default) if default is not None else ':')) - elif (int(Byte) / OneKBUnit) > 0.99: # MB - MB = int(Byte) / OneKBUnit - if Color is True: - return (clr.warnye + "%7.1f" % round(MB, 1) + 'K' + clr.end) + if ans in ['', None]: + if type in ['host']: + ans = default + + if ans: + if '"' in ans or '\\' in ans: + print(" %s is invalid character" % ans) + continue + if type == int: + if ans.isdigit(): + return int(ans) + else: + print('Invalid Digit type') + elif type == 'ip': + if IsIpValid(ans): + return ans + else: + print('Invalid Ip Type') + elif type == 'host': + if IsIpValid(ans): + ret, hostname, ip = GetHostInfo(ip=ans) + else: + ret, hostname, ip = GetHostInfo(hostname=ans) + + if ret is False: + print('Invalid Host') + continue + else: + return ip + + elif type == 'url': + if IsUrlValid(ans): + return ans + else: + print('Invalid Url Type') + else: + if ValidAnsList is not None: + if ans not in ValidAnsList: + print('Invalid Answer Type. Only %s is supported' % ', '.join(ValidAnsList)) + continue + else: + if ans.lower() == 'null': + return '' + + return ans else: - return "%7.1f" % round(MB, 1) + 'K' - else: # KB - KB = int(Byte) - if Color is True: - return (clr.badre + "%7.1f" % round(KB, 1) + 'B' + clr.end) + if default == '': + if type in ['pwd', 'net_device', 'host']: + print('Invalid %s type' % type) + continue + + if default is not None: + return default + + return ans + except Exception as err: + print(err, sys.exc_info()[2].tb_lineno) + + + +def isValidConfig(ConfPath, *CheckKeys, **kwargs): + """ + check if conf is valid with CheckKey + :param ConfPath: file path + :param **kwargs: check if the value is valid with CheckKeys + :return: bool + """ + + ret, conf = GetConf(ConfPath) + if ret is False: + return ret, conf + else: + isValid = True + for key in CheckKeys: + if conf.mgs.__dict__[key] is None or conf.mgs.__dict__[key] == '': + isValid = False + + for key, val in kwargs: + if conf.mgs.__dict__[key] != val: + isValid = False + + return isValid, conf + + + +def UpdateConf(Path, Section, Key, Value, logger, Force=True): + """ + Update NetowrkId and ServerId in ksanMonitor.conf + :param Path: + :param Section: + :param Key: + :param Value: + :param Force: + :return: + """ + res, conf = read_conf(Path) + if res is True: + if Section not in conf: + logger.error('Invalid Section: %s in %s' % (Section, Path)) + return False + if Key not in conf[Section] and Force is False: + logger.error('Invalid Key: %s in %s' % (Key, Path)) + return False + conf[Section][Key] = Value + config = ConfigParser() + config.optionxform = str + for section in conf.keys(): + config[section] = dict() + for key, val in conf[section].items(): + config[section][key] = val + + with open(Path, 'w') as configfile: + config.write(configfile) + return True + else: + return False + + + +@catch_exceptions() +class ResPonseHeader(object): + def __init__(self, dic, logger=None): + try: + self.IsNeedLogin = dic['IsNeedLogin'] + self.AccessDenied = dic['AccessDenied'] + self.Result = dic['Result'] + self.Code = dic['Code'] + self.Message = dic['Message'] + if 'Data' in dic: + self.Data = dic['Data'] + except KeyError as err: + if logger is not None: + logger.error(err, sys.exc_info()[2].tb_lineno) else: - return "%7.1f" % round(KB, 1) + 'B' - elif Title == 'DiskRw': - if (int(Byte) / OneTBUnit) > 0.99: # TB - TB = int(Byte) / OneTBUnit - if Color is True: - return (clr.okbl + "%5.1f" % round(TB, 1) + 'T' + clr.end) + print(err, sys.exc_info()[0].tb_lineno) + + +##### Items Header of Response Body Class ##### +class ResPonseItemsHeader(object): + def __init__(self, dic, logger=None): + try: + self.TotalCount = dic['TotalCount'] + self.Skips = dic['Skips'] + self.PageNo = dic['PageNo'] + self.CountPerPage = dic['CountPerPage'] + self.PagePerSection = dic['PagePerSection'] + self.TotalPage = dic['TotalPage'] + self.StartPageNo = dic['StartPageNo'] + self.EndPageNo = dic['EndPageNo'] + self.PageNos = dic['PageNos'] + self.HavePreviousPage = dic['HavePreviousPage'] + self.HaveNextPage = dic['HaveNextPage'] + self.HavePreviousPageSection = dic['HavePreviousPageSection'] + self.HaveNextPageSection = dic['HaveNextPageSection'] + self.Items = dic['Items'] + except KeyError as err: + if logger is not None: + logger.error(err, sys.exc_info()[2].tb_lineno) else: - return "%5.1f" % round(TB, 1) + 'T' - elif (int(Byte) / OneGBUnit) > 0.99: # GB - GB = int(Byte) / OneGBUnit - if Color is True: - return (clr.okbl + "%5.1f" % round(GB, 1) + 'G' + clr.end) + print(err, sys.exc_info()[0].tb_lineno) + + +@catch_exceptions() +class GetApiResult: + def __init__(self, Header, ItemHeader, Items): + self.Header = Header + self.ItemHeader = ItemHeader + self.Items = Items + + +#AuthKey = "5de46d7ccd5d0954fad7d11ffc22a417e2784cbedd9f1dae3992a46e97b367e8" +IfsPortalIp = None +IfsPortalPort = None + +# http response status +CONTINUE = 100 +SWITCHING_PROTOCOLS = 101 +PROCESSING = 102 +OK = 200 +CREATED = 201 +ACCEPTED = 202 +NON_AUTHORITATIVE_INFORMATION = 203 +NO_CONTENT = 204 +RESET_CONTENT = 205 +PARTIAL_CONTENT = 206 +MULTI_STATUS = 207 +IM_USED = 226 +MULTIPLE_CHOICES = 300 +MOVED_PERMANENTLY = 301 +FOUND = 302 +SEE_OTHER = 303 +NOT_MODIFIED = 304 +USE_PROXY = 305 +TEMPORARY_REDIRECT = 307 +BAD_REQUEST = 400 +UNAUTHORIZED = 401 +PAYMENT_REQUIRED = 402 +FORBIDDEN = 403 +NOT_FOUND = 404 +METHOD_NOT_ALLOWED = 405 +NOT_ACCEPTABLE = 406 +PROXY_AUTHENTICATION_REQUIRED = 407 +REQUEST_TIMEOUT = 408 +CONFLICT = 409 +GONE = 410 +LENGTH_REQUIRED = 411 +PRECONDITION_FAILED = 412 +REQUEST_ENTITY_TOO_LARGE = 413 +REQUEST_URI_TOO_LONG = 414 +UNSUPPORTED_MEDIA_TYPE = 415 +REQUESTED_RANGE_NOT_SATISFIABLE = 416 +EXPECTATION_FAILED = 417 +UNPROCESSABLE_ENTITY = 422 +LOCKED = 423 +FAILED_DEPENDENCY = 424 +UPGRADE_REQUIRED = 426 +PRECONDITION_REQUIRED = 428 +TOO_MANY_REQUESTS = 429 +REQUEST_HEADER_FIELDS_TOO_LARGE = 431 +INTERNAL_SERVER_ERROR = 500 +NOT_IMPLEMENTED = 501 +BAD_GATEWAY = 502 +SERVICE_UNAVAILABLE = 503 +GATEWAY_TIMEOUT = 504 +HTTP_VERSION_NOT_SUPPORTED = 505 +INSUFFICIENT_STORAGE = 507 +NOT_EXTENDED = 510 +NETWORK_AUTHENTICATION_REQUIRED = 511 + + +class RestApi: + + def __init__(self, ip, port, url, header=None, params=None, protocol='https', authkey=None, logger=None): + self._ip = ip + self._port = port + self._params = params + self._protocol = protocol + self._url = url + self.logger = logger + if self.logger is None: + self.logger = Logging().get_logger() + + if header is None: + header = dict() + header[HeaderAuth] = authkey + + self._header = header + if params is None: + params = dict() + self._params = params + + def get(self, ItemsHeader=True, ReturnType=None): + url = "%s://%s:%d%s" % (self._protocol, self._ip, self._port, self._url) + urllib3.disable_warnings() + err_msg = '' + try: + r = requests.get(url, params=self._params, verify=False, headers=self._header) + self.logging_request_info(GetMethod, r) + ret = r.content.decode(UTF8) + Data = json.loads(ret) + Ret = self.GetResponse(Data,ItemsHeader=ItemsHeader, ReturnType=ReturnType) + return ResOk, '', Ret + except requests.ConnectionError as err: + self.logger.error("Connection Error %s" % err) + return ResConnectionErrorCode, ResConnectionErrorMsg,None + except requests.URLRequired as err: + self.logger.error("UrlRequired Error %s" % err) + return ResInvalidCode, ResInvalidMsg, None + except (requests.ConnectTimeout, requests.ReadTimeout, requests.Timeout) as err: + self.logger.error("Timeout Error %s" % err) + return ResTimeErrorCode, ResTimeErrorCodeMsg, None + except Exception as err: + self.logger.error("Other Error %s" % err) + return ResEtcErrorCode, ResEtcErrorMsg, None + + def post(self,ItemsHeader=True, ReturnType=None): + url = "%s://%s:%d%s" % (self._protocol, self._ip, self._port, self._url) + urllib3.disable_warnings() + try: + self._header[HeaderContentType] = 'application/json' + # self._params = json.dumps(self._params) + r = requests.post(url=url, data=self._params, headers=self._header, verify=False) + self.logging_request_info(PostMethod, r) + ret = r.content.decode(UTF8) + Data = json.loads(ret) + Ret = self.GetResponse(Data,ItemsHeader=ItemsHeader, ReturnType=ReturnType) + return ResOk, '', Ret + + except requests.ConnectionError as err: + self.logger.error("Connection Error %s" % err) + return ResConnectionErrorCode, ResTimeErrorCodeMsg,None + except requests.URLRequired as err: + self.logger.error("UrlRequired Error %s" % err) + return ResInvalidCode, ResInvalidMsg, None + except (requests.ConnectTimeout, requests.ReadTimeout, requests.Timeout) as err: + self.logger.error("Timeout Error %s" % err) + return ResTimeErrorCode, ResTimeErrorCodeMsg, None + except Exception as err: + self.logger.error("Other Error %s" % err) + return ResEtcErrorCode, ResEtcErrorMsg, None + + def put(self,ItemsHeader=True, ReturnType=None): + url = "%s://%s:%d%s" % (self._protocol, self._ip, self._port, self._url) + urllib3.disable_warnings() + try: + self._header[HeaderContentType] = 'application/json-patch+json' + #self._params = json.dumps(self._params) + r = requests.put(url=url, data=self._params, headers=self._header, verify=False) + self.logging_request_info(PutMethod, r) + ret = r.content.decode(UTF8) + Data = json.loads(ret) + Ret = self.GetResponse(Data,ItemsHeader=ItemsHeader, ReturnType=ReturnType) + return ResOk, '', Ret + except requests.ConnectionError as err: + self.logger.error("Connection Error %s" % err) + return ResConnectionErrorCode, ResTimeErrorCodeMsg,None + except requests.URLRequired as err: + self.logger.error("UrlRequired Error %s" % err) + return ResInvalidCode, ResInvalidMsg, None + except (requests.ConnectTimeout, requests.ReadTimeout, requests.Timeout) as err: + self.logger.error("Timeout Error %s" % err) + return ResTimeErrorCode, ResTimeErrorCodeMsg, None + except Exception as err: + self.logger.error("Other Error %s" % err) + return ResEtcErrorCode, ResEtcErrorMsg, None + + @catch_exceptions() + def delete(self, ItemsHeader=True, ReturnType=None): + url = "%s://%s:%d%s" % (self._protocol, self._ip, self._port, self._url) + urllib3.disable_warnings() + try: + self._header[HeaderContentType] = 'application/json-patch+json' + #self._params = json.dumps(self._params) + r = requests.delete(url=url, data=self._params, headers=self._header, verify=False) + self.logging_request_info(DeleteMethod, r) + ret = r.content.decode(UTF8) + Data = json.loads(ret) + Ret = self.GetResponse(Data,ItemsHeader=ItemsHeader, ReturnType=ReturnType) + return ResOk, '', Ret + + except requests.ConnectionError as err: + self.logger.error("Connection Error %s" % err) + return ResConnectionErrorCode, ResTimeErrorCodeMsg,None + except requests.URLRequired as err: + self.logger.error("UrlRequired Error %s" % err) + return ResInvalidCode, ResInvalidMsg, None + except (requests.ConnectTimeout, requests.ReadTimeout, requests.Timeout) as err: + self.logger.error("Timeout Error %s" % err) + return ResTimeErrorCode, ResTimeErrorCodeMsg, None + except Exception as err: + self.logger.error("Other Error %s" % err) + return ResEtcErrorCode, ResEtcErrorMsg, None + @catch_exceptions() + def logging_request_info(self, method, req): + try: + self.logger.debug("Request %s url >>>>> %s" % (method, req.url)) + self.logger.debug("Header : %s" % req.request.headers) + self.logger.debug("Body : %s" % req.request.body) + except Exception as err: + print(err) + + def logging_response_info(self, Data): + try: + self.logger.debug("Response Header <<<<< %s" % str(Data)) + except Exception as err: + print(err) + + @catch_exceptions() + def GetResponse(self, Ret, ItemsHeader=False, ReturnType=None): + """ + Convert Dict Result to ResponseHeader Class + There are three kinds of ResponseHeader + - single type1(data:True/False) + - sigle type2() + - multi value(dict)) + """ + self.logging_response_info(Ret) + if 'Data' in Ret: + Ret.update({"py/object": ResponseHeaderWithDataModule}) + if Ret['Data'] is None: # if data items not exists. + Ret = jsonpickle.decode(json.dumps(Ret)) + return Ret + + if ItemsHeader is True: # if items header exists + Ret['Data'].update({"py/object": ResponseItemsHeaderModule}) + for Item in Ret['Data']['Items']: + DeserializeResponseReculsive(Item, ReturnType) else: - return "%5.1f" % round(GB, 1) + 'G' - elif (int(Byte) / OneMBUnit) > 0.99: # MB - MB = int(Byte) / OneMBUnit - if Color is True: - return (clr.okgr + "%5.1f" % round(MB, 1) + 'M' + clr.end) + DeserializeResponseReculsive(Ret['Data'], ReturnType) + + else: + # Return + DeserializeResponseReculsive(Ret, ResponseHeaderModule) + + Ret = jsonpickle.decode(json.dumps(Ret)) + return Ret + + def GetResponse1(self, Data, ItemsHeader=False): + self.logging_response_info(Data) + Data.update({"py/object": ResponseHeaderModule}) + if ItemsHeader is True: + Data['Data'].update({"py/object": ResponseItemsHeaderModule}) + + Ret = jsonpickle.decode(json.dumps(Data)) + return Ret + + @catch_exceptions() + def parsing_result(self, header): + ''' + cast total result dict to class(header, item header, data) + :param header: dict + :return: GetApiResult Object + ''' + Header = None + ItemsHeader = None + Data = None + try: + self.logger.debug("Response << %s" % str(header)) + Header = ResPonseHeader(header) + if "Data" in Header.__dict__: + if isinstance(Header.Data, dict): + if "Items" in Header.Data: + ItemsHeader = ResPonseItemsHeader(Header.Data) + Data = DictToObject(Header.Data) + else: + Data = Header.Data + # print header attr + self.logger.debug("Header: %s" % str(Header.__dict__)) + + # print ItemsHeader attr + if hasattr(ItemsHeader, '__dict__'): + dict_attr = str(ItemsHeader.__dict__) else: - return "%5.1f" % round(MB, 1) + 'M' + dict_attr = 'None' + self.logger.debug("ItemsHeader: %s" % dict_attr) + + # print Data attr - elif (int(Byte) / OneKBUnit) > 0.99: # MB - MB = int(Byte) / OneKBUnit - if Color is True: - return (clr.warnye + "%5.1f" % round(MB, 1) + 'K' + clr.end) + if hasattr(Data, '__dict__'): + dict_attr = str(Data.__dict__) else: - return "%5.1f" % round(MB, 1) + 'K' - else: # KB - KB = int(Byte) - if Color is True: - return (clr.badre + "%5.1f" % round(KB, 1) + 'B' + clr.end) + dict_attr = 'None' + self.logger.debug("Data: %s" % dict_attr) + + except Exception as err: + self.logger.error("fail to parsing response data: %s line:%d" % (str(err), sys.exc_info()[2].tb_lineno)) + finally: + return GetApiResult(Header, ItemsHeader, Data) + + +@catch_exceptions() +def DeserializeResponse(Data, ObjectType): + """ + Get Data parsing with ObjectType Class + :param Data: Dict type, + :param ObjectType: Class name + :return: ObjectType class + """ + Data.update({"py/object": ObjectType}) + Ret = jsonpickle.decode(json.dumps(Data)) + return Ret + + +def DeserializeResponseReculsive(Data, ObjectType): + """ + Get Data parsing with ObjectType Class + :param Data: Dict type, + :param ObjectType: Class name + :return: ObjectType class + """ + if ObjectType is not None: + Data.update({"py/object": ObjectType}) + for key, val in Data.items(): + if key in Parsing.keys(): + if isinstance(val, list): + for val1 in Data[key]: + DeserializeResponseReculsive(val1, Parsing[key]) else: - return "%5.1f" % round(KB, 1) + 'B' + if isinstance(val, dict): + Data[key].update({"py/object": Parsing[key]}) -def DisplayDiskState(State): - if State == DiskStart: - return (clr.okgr + "%7s" % State + clr.end) - elif State == DiskStop: - return (clr.badre + "%7s" % State + clr.end) - elif State == DiskWeak: - return (clr.warnye + "%7s" % State + clr.end) - elif State == DiskDisable: - return (clr.badre + "%7s" % State + clr.end) +def get_res(res): -def DisplayState(State): - if State in [ServiceStateOnline, ServerStateOnline]: - return (clr.okgr + "%10s" % State + clr.end) - elif State in [ServerStateOffline, ServerStateTimeout, ServiceStateOffline, ServiceStateUnkown]: - return (clr.badre + "%10s" % State + clr.end) + print("IsNeedLogin:", res.IsNeedLogin) + print("AccessDenied:", res.AccessDenied) + print("Result:", res.Result) + print("Code:", res.Code) + print("Message:", res.Message) + if 'Data' in res.__dict__: + if not isinstance(res.Data, dict): + print("Data:", res.Data) -def DisplayDiskMode(Mode): - if Mode == DiskModeRw: - return DiskModeRwShort - else: - return DiskModeRoShort +class GetNetwork(object): + def __init__(self, Ip=None, logger=None): + self.logger = logger + self.NicInfo = None + self.nicinfo_list = list() + self.GetNicInfo(Ip=Ip) + self.IoCounterPerNic = psutil.net_io_counters(pernic=True) + + @catch_exceptions() + def GetNicInfo(self, Ip=None): + """ + get all detail nic info list. + :return: + """ + # get dns info + dnsresolvers = dns.resolver.Resolver() + # get status of each nic(isup(tru/false), duplex, speed, mtu) + netif_stat = psutil.net_if_stats() + # get nic info. netif_addresses includes {'eno1':..., 'eno2':...} + netif_addresses = psutil.net_if_addrs() + for nic in netif_addresses.keys(): + if nic == 'lo' or netifaces.AF_INET not in netifaces.ifaddresses(nic): + continue + tmp_dic = dict() + tmp_dic['Name'] = nic + tmp_dic['Dhcp'] = 'No' # must 'No' + tmp_dic['MacAddress'] = netifaces.ifaddresses(nic)[netifaces.AF_LINK][0]['addr'] + tmp_dic['LinkState'] = 'Up' if netif_stat[nic].isup is True else 'Down' + tmp_dic['IpAddress'] = netifaces.ifaddresses(nic)[netifaces.AF_INET][0]['addr'] + tmp_dic['SubnetMask'] = netif_addresses[nic][0].netmask + tmp_dic['BandWidth'] = netif_stat[nic].speed + tmp_dic['Gateway'] = self.GetGatewayWithNic(nic) + tmp_dic['Dns1'] = '' + tmp_dic['Dns2'] = '' + for idx, nameserver in enumerate(dnsresolvers.nameservers, start=1): + key = 'Dns%d' % idx + tmp_dic[key] = nameserver + if tmp_dic['Dns2'] == '': + tmp_dic['Dns2'] = tmp_dic['Dns1'] + + + self.nicinfo_list.append(tmp_dic) + if Ip is not None: + if Ip == tmp_dic['IpAddress']: + self.NicInfo = tmp_dic + return tmp_dic + + @catch_exceptions() + def GetGatewayWithNic(self, NicName): + gw = netifaces.gateways() + for gateway, nic, is_default in gw[netifaces.AF_INET]: + if NicName == nic: + return gateway + return '' + + + @catch_exceptions() + def GetNicInfoWithNicName(self, NicName): + """ + return specific nic info with nic name + :return: success to get nic info {'name', , 'Dhcp': , 'MacAddress':, 'LinkState': , + 'addr',
, 'netask': , 'gateway': , 'Dns1': , 'Dns2': , 'Dns3':...}, fail to get nic info: None + """ + if NicName is None: + print('nic is None') + else: + for nic in self.nicinfo_list: + if nic['Name'] == NicName: + return nic + return None + + + def ReNewalNicStat(self): + """ + Update Nic stat + :return: + """ + self.IoCounterPerNic = psutil.net_io_counters(pernic=True) + self.GetNicInfo() + + + + +class Daemon(object): + def __init__(self, PidFile, DaemonName): + self.PidFile = PidFile + self.DaemonName = DaemonName + self.redirect = '/dev/null' + self.pid = -1 + + def daemonize(self): + try: + pid = os.fork() + if pid > 0: + # exit first parent + sys.exit(0) + except OSError as e: + sys.stderr.write("fork failed: %d (%s)\n" % (e.errno, e.strerror)) + sys.exit(1) + + # decouple from parent environment + os.chdir("/") + os.setsid() + os.umask(0) + + # do second fork + try: + pid = os.fork() + if pid > 0: + # exit from second parent + sys.exit(0) + except OSError as e: + self.pid = pid + sys.stderr.write("fork failed: %d (%s)\n" % (e.errno, e.strerror)) + sys.exit(1) + + # write pidfile + atexit.register(self.delpid) + self.pid = str(os.getpid()) + if self.PidFile: + with open(self.PidFile, 'w+') as f: + f.write("%s" % self.pid) + + # redirect standard file descriptors + sys.stdout.flush() + sys.stderr.flush() + sio = open(self.redirect, 'a+') + os.dup2(sio.fileno(), sys.stdin.fileno()) + os.dup2(sio.fileno(), sys.stdout.fileno()) + os.dup2(sio.fileno(), sys.stderr.fileno()) + + sio.close() + + def delpid(self): + if self.PidFile and os.path.exists(self.PidFile): + os.remove(self.PidFile) + + def start(self): + # Start the daemon + self.daemonize() + self.run() + + def stop(self): + Ret, Pid = IsDaemonRunning(self.PidFile, self.DaemonName) + if Ret is False: + return + try: + while 1: + os.kill(int(Pid), signal.SIGKILL) + time.sleep(0.1) + except OSError as err: + err = str(err) + if err.find("No such process") > 0: + if os.path.exists(self.PidFile): + os.remove(self.PidFile) + else: + sys.exit(1) + def restart(self): + self.stop() + self.start() + def run(self): + sys.stderr.write("run:\n") + pass diff --git a/core/common/configure/config.py b/core/common/configure/config.py index cdacde9b..8b11bf00 100644 --- a/core/common/configure/config.py +++ b/core/common/configure/config.py @@ -9,9 +9,7 @@ * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. """ -from const.common import * -from service.service_manage import SetServiceConfig, GetServiceConfig, DsPServiceConf, \ - UpdateServiceConfigVersion, RemoveServiceConfig, GetServiceConfigList, ShowConfigList +from portal_api.apis import * def ConfigUtilHandler(Conf, Action, Parser, logger): @@ -61,8 +59,16 @@ def ConfigUtilHandler(Conf, Action, Parser, logger): Parser.print_help() print('config version is required.') sys.exit(-1) + Version = options.ConfigVersionId ServiceType = ServiceTypeConversion[options.ServiceType.lower()] + if str(Version).lower() == 'last': + LatestConfig, ErrLog = GetLatestServiceConfig(PortalIp, PortalPort, PortalApiKey, ServiceType, logger=logger) + if LatestConfig is None: + print('fail to get last version config info. %s' % ErrLog) + return + Version = LatestConfig.Version + Res, Errmsg, Ret = UpdateServiceConfigVersion(PortalIp, PortalPort, PortalApiKey, ServiceType, Version, logger=logger) print(Ret.Result, Ret.Message) diff --git a/core/common/const/common.py b/core/common/const/common.py index 47d118b9..27fec35b 100644 --- a/core/common/const/common.py +++ b/core/common/const/common.py @@ -11,8 +11,15 @@ import os, sys if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.log import catch_exceptions from pydantic import BaseModel +import re +import psutil +import time +import inspect +import jsonpickle +import json +import threading +from optparse import OptionParser """ ##### UNIT ##### @@ -27,24 +34,6 @@ ######### environment configuration define ######## """ -### Conf Key ### -KeyCommonSection = 'mgs' -KeyPortalHost = 'PortalHost' -KeyPortalPort = 'PortalPort' -KeyPortalApiKey = 'PortalApiKey' -KeyMQHost = 'MQHost' -KeyMQPort = 'MQPort' -KeyMQUser = 'MQUser' -KeyMQPassword = 'MQPassword' -KeyServerId = 'ServerId' -KeyManagementNetDev = 'ManagementNetDev' -KeyDefaultNetworkId = 'DefaultNetworkId' - -KeyMonitorSection = 'monitor' -KeyServerMonitorInterval = 'ServerMonitorInterval' -KeyNetworkMonitorInterval = 'NetworkMonitorInterval' -KeyDiskMonitorInterval = 'DiskMonitorInterval' -KeyServiceMonitorInterval = 'ServiceMonitorInterval' Updated = 1 # disk, network, service, server changed flag @@ -75,106 +64,8 @@ OptDiskPoolName2 = '--diskpoolname' -##### CONFIG FILE TYPE ##### -ConfigTypeINI = 'INI' -ConfigTypeObject = 'Object' - -##### display option ###### -MoreDetailInfo = 'MoreDetail' -DetailInfo = 'Detail' -SimpleInfo = 'Simple' - -##### PATH ##### -KsanEtcPath = '/usr/local/ksan/etc' -KsanBinPath = '/usr/local/ksan/bin' -KsanSbinPath = '/usr/local/ksan/sbin' -KsanSslPath = '/usr/local/ksan/ssl' -KsanUtilDirPath = '/usr/local/ksan/bin/util' - -MonServicedConfPath = '/usr/local/ksan/etc/ksanAgent.conf' -DiskPoolXmlPath = '/usr/local/ksan/etc/diskpools.xml' -ServicePoolXmlPath = '/var/log/ksan/agent/servicepools_dump.xml' -OsdXmlFilePath = '/usr/local/ksan/etc/ksan-osd-log.xml' -GwXmlFilePath = '/usr/local/ksan/etc/ksan-gw-log.xml' -KsanMongDbManagerBinPath = '/usr/local/ksan/bin/ksanMongoDBManager' - -ProcDiskStatsPath = '/proc/diskstats' - -##### ksanMonitor.conf ##### -DiskMonitorInterval = 10 -ServerMonitorInterval = 10 -ServiceMonitorInterval = 10 -NetworkMonitorInterval = 10 -IntervalShort = 1 -IntervalMiddle = 5 -IntervalLong = 10 -### service action retry count ### -ServiceContolRetryCount = 3 -""" -##### HTTP ##### -""" -### http header key ### -HeaderContentType = 'Content-Type' -HeaderAuth = 'Authorization' - - -### http method ### -GetMethod = 'GET' -PostMethod = 'POST' -PutMethod = 'PUT' -DeleteMethod = 'DELETE' - -##### return code & messages define ##### -RetKeyResult = 'Result' -RetKeyCode = 'Code' -RetKeyMessage = 'Message' - - - -ResOk = 0 -ResNotFoundCode = 2 -ResNotFoundMsg = 'Not found ' -ResInvalidCode = 22 -ResInvalidMsg = 'Invalid Error ' -ResConnectionErrorCode = 111 -ResConnectionErrorMsg ='Connection Error ' -ResTimeErrorCode = 11 -ResTimeErrorCodeMsg = 'Timeout Error ' -ResDuplicateCode = 17 -ResDuplicateMsg = 'Duplicated' -ResDiskAlreadyExists = 'Disk Id already exists. Initialize first' - -ResEtcErrorCode = 1 -ResEtcErrorMsg = 'Other Error ' -ResFailToGetVlainId = 'Fail to get Vlan Id' - -ResultFail = 1 # fail to get data - -ResultSuccess = 'Success' -ResultWarning = 'Warning' -ResultError = 'Error' -CodeSuccess = 'EC000' -MessageNull = '' - -CodeFailServerInit = 'EC001' -MessageFailServerInit = 'Fail to init Server' - - -CodeDuplicated = 'EC036' -ResutlNotFound = 'EC014' - -ErrMsgServerNameMissing = 'ServerName is required' -ErrMsgDiskNameMissing = 'DiskName is required' -ErrMsgDiskpoolNameMissing = 'DiskpoolName is required' -ErrMsgUserNameMissing = 'UserName is required' - - - -##### ENCODING ##### -UTF8 = 'utf-8' - """ ##### RABBITMQ ##### """ @@ -206,23 +97,17 @@ DiskPoolReplica1 = 'OnePlusZero' DiskPoolReplica2 = 'OnePlusOne' DiskPoolReplica3 = 'OnePlusTwo' +DiskPoolReplicaEC = 'ErasureCode' +ECValueParser = re.compile("ec\(([\d]+):([\d]+)\)") + ### diskpool type ### DiskPoolClassStandard = 'STANDARD' DiskPoolClassArchive = 'ARCHIVE' +DiskPoolClassPerformance = 'PERFORMANCE' -##### Conversion Dict to Object Class ##### -@catch_exceptions() -class DictToObject(object): +ValidDiskPoolType = [DiskPoolClassStandard, DiskPoolClassArchive, DiskPoolClassPerformance] - def __init__(self, myDict): - for key, value in myDict.items(): - if type(value) == dict: - setattr(self, key, DictToObject(value)) - else: - if isinstance(value, str) and value.isdigit(): - value = int(value) - setattr(self, key, value) """ ##### SERVER ##### @@ -236,21 +121,62 @@ def __init__(self, myDict): """ ##### SERVICE ##### """ + +### service name define ### +KsanApiPortalName = 'ksanApiPortal' +KsanPortal = 'ksanPortal' +KsanPortalBridgeName = 'ksanPortalBridge' +KsanAgentName = 'ksanAgent' +KsanGWName = 'ksanGW' +KsanOSDName = 'ksanOSD' +KsanLifecycleManagerName = 'ksanLifecycleManager' +KsanReplicationManagerName = 'ksanReplicationManager' +KsanLogManagerName = 'ksanLogManager' +KsanRecoveryName = 'KsanRecovery' + +MongoDBName = 'MongoDB' +MariaDBName = 'MariaDB' +RabbitMQName = 'RabbitMQ' + + +### docker contailner name define ### +KsanOSDContainerName = 'ksan-osd' +KsanGWContainerName = 'ksan-gw' +KsanLogManagerContainerName = 'ksan-log-manager' +KsanLifecycleManagerContainerName = 'ksan-lifecycle-manager' +KsanReplicationMangerContainerName = 'ksan-replication-manager' +KsanApiPortalContainerName = 'ksan-api-portal' +KsanPortalContainerName = 'ksan-portal' +KsanPortalBridgeContainerName= 'ksan-portal-bridge' + + ### service binamry name ### -KsanOsdBinaryName = 'ksan-osd.jar' -KsanGwBinaryName = 'ksan-gw.jar' +KsanOsdBinaryName = '%s.jar' % KsanOSDName +KsanGwBinaryName = '%s.jar' % KsanGWName KsanRecoveryBinaryName = 'ksanRecovery' KsanMonitorBinaryName = 'ksanMonitor' KsanAgentBinaryName = 'ksanAgent' KsanUtilName = 'ksan' KsanServerRegister = 'ksanServerRegister' -KsanUtilPath = '%s/ksan' % KsanBinPath -KsanAgentPath = '%s/%s' % (KsanSbinPath, KsanAgentBinaryName) -KsanGwPath = '%s/%s' % (KsanSbinPath, KsanGwBinaryName) -KsanOsdPath = '%s/%s' % (KsanSbinPath, KsanOsdBinaryName) -KsanRecoveryPath = '%s/%s' % (KsanSbinPath, KsanRecoveryBinaryName) -KsanServerRegisterPath = '%s/%s' % (KsanUtilDirPath, KsanServerRegister) + + +### service type ### +TypeServiceOSD = KsanOSDName +TypeServiceGW = KsanGWName +TypeServiceMongoDB = 'MongoDB' +TypeServiceMariaDB = 'MariaDB' +TypeServiceS3Backend = 'S3Backend' +TypeServiceMonitor = 'ksanMonitor' +TypeServiceAgent = KsanAgentName +TypeServiceRabbitMq = 'RabbitMQ' +TypeServiceFsck = 'ksanFsck' +TypeServiceGetAttr = 'ksanGetAttr' +TypeServiceCbalance = 'ksanCbalance' +TypeServiceLifecycle = KsanLifecycleManagerName +TypeServiceRecovery = 'ksanRecovery' +TypeServiceReplication = KsanReplicationManagerName +TypeServiceLogManager = KsanLogManagerName ### service state ### @@ -259,12 +185,7 @@ def __init__(self, myDict): ServiceStateUnkown = 'Unknown' ### service pid file ### - -KsanOsdPidFile = '/var/run/ksanOsd.pid' -KsanGwPidFile = '/var/run/ksanGw.pid' -KsanMonitorPidFile = '/var/run/ksanMonitor.pid' KsanAgentPidFile = '/var/run/ksanAgent.pid' -KsanMongosPidFile = '/var/run/mongod.pid' ### service status ### @@ -275,38 +196,17 @@ def __init__(self, myDict): OFFLINE = 'Offline' -### service type ### -TypeServiceOSD = 'ksanOSD' -TypeServiceGW = 'ksanGW' -TypeServiceMongoDB = 'MongoDB' -TypeServiceMariaDB = 'MariaDB' -TypeServiceObjmanager = 'OBJMANAGER' -TypeServiceS3Backend = 'S3Backend' -TypeServiceMonitor = 'ksanMonitor' -TypeServiceAgent = 'ksanAgent' -TypeServiceRabbitMq = 'RabbitMq' -TypeServiceHaproxy = 'HaProxy' -TypeServiceFsck = 'ksanFsck.jar' -TypeServiceGetAttr = 'ksanGetAttr.jar' -TypeServiceCbalance = 'ksanCbalance.jar' -TypeServiceLifecycle = 'ksanLifecycle' -TypeServiceRecovery = 'ksanRecovery' -SampleS3ConfFile = './objmanager.conf' -S3ConfFile = '/opt/objmanager.conf' -SampleHaproxyConfFile = './haproxy.cfg' -HaproxyConfFile = '/opt/haproxy.cfg' -ServiceStart = 'Start' -ServiceStop = 'Stop' -ServiceRestart = 'Restart' ### service id path define ### -KsanAgentServiceIdHiddenPath = '/usr/local/ksan/sbin/.ksanAgent.ServiceId' -KsanOSDServiceIdHiddenPath = '/usr/local/ksan/sbin/.ksanOSD.ServiceId' -KsanGWServiceIdHiddenPath = '/usr/local/ksan/sbin/.ksanGW.ServiceId' -KsanLifecycleServiceIdHiddenPath = '/usr/local/ksan/sbin/.ksanLifecycle.ServiceId' +KsanAgentServiceIdHiddenPath = '/usr/local/ksan/sbin/.%s.ServiceId' % KsanAgentName +KsanOSDServiceIdHiddenPath = '/usr/local/ksan/sbin/.%s.ServiceId' % KsanOSDName +KsanGWServiceIdHiddenPath = '/usr/local/ksan/sbin/.%s.ServiceId' % KsanGWName +KsanLifecycleServiceIdHiddenPath = '/usr/local/ksan/sbin/.%s.ServiceId' % KsanLifecycleManagerName KsanRecoveryServiceIdHiddenPath = '/usr/local/ksan/sbin/.ksanRecovery.ServiceId' +KsanReplicationServiceIdHiddenPath = '/usr/local/ksan/sbin/.%s.ServiceId' % KsanReplicationManagerName +KsanLogManagerServiceIdHiddenPath = '/usr/local/ksan/sbin/.%s.ServiceId' % KsanLogManagerName ServiceTypeServiceHiddenPathMap = dict() ServiceTypeServiceHiddenPathMap[TypeServiceAgent] = KsanAgentServiceIdHiddenPath @@ -314,41 +214,176 @@ def __init__(self, myDict): ServiceTypeServiceHiddenPathMap[TypeServiceGW] = KsanGWServiceIdHiddenPath ServiceTypeServiceHiddenPathMap[TypeServiceLifecycle] = KsanLifecycleServiceIdHiddenPath ServiceTypeServiceHiddenPathMap[TypeServiceRecovery] = KsanRecoveryServiceIdHiddenPath +ServiceTypeServiceHiddenPathMap[TypeServiceReplication] = KsanReplicationServiceIdHiddenPath +ServiceTypeServiceHiddenPathMap[TypeServiceLogManager] = KsanLogManagerServiceIdHiddenPath + ### service unit ### -SystemdKsanGWServiceName = 'ksangw.service' -SystemdKsanOSDServiceName = 'ksanosd.service' -SystemdKsanAgentServiceName = 'ksanagent.service' -SystemdKsanLifecycleServiceName = 'ksanlifecycle.service' -SystemdKsanRecoveryServiceName = 'ksanrecovery.service' +SystemdKsanGWServiceName = '%s.service' % KsanGWName +SystemdKsanOSDServiceName = '%s.service' % KsanOSDName +SystemdKsanAgentServiceName = '%s.service' % KsanAgentName +SystemdKsanLifecycleServiceName = '%s.service' % KsanLifecycleManagerName +SystemdKsanRecoveryServiceName = '%s.service' % KsanRecoveryName +SystemdKsanReplicationServiceName = '%s.service' % KsanReplicationManagerName +SystemdKsanLogManagerServiceName = '%s.service' % KsanLogManagerName + + ServiceTypeSystemdServiceMap = dict() ServiceTypeSystemdServiceMap[TypeServiceGW] = SystemdKsanGWServiceName ServiceTypeSystemdServiceMap[TypeServiceOSD] = SystemdKsanOSDServiceName ServiceTypeSystemdServiceMap[TypeServiceLifecycle] = SystemdKsanLifecycleServiceName ServiceTypeSystemdServiceMap[TypeServiceRecovery] = SystemdKsanRecoveryServiceName +ServiceTypeSystemdServiceMap[TypeServiceReplication] = SystemdKsanReplicationServiceName +ServiceTypeSystemdServiceMap[TypeServiceLogManager] = SystemdKsanLogManagerServiceName ### docker container servicename map ### ServiceTypeDockerServiceContainerNameMap = dict() -ServiceTypeDockerServiceContainerNameMap[TypeServiceOSD] = 'ksan-osd' -ServiceTypeDockerServiceContainerNameMap[TypeServiceGW] = 'ksan-gw' -ServiceTypeDockerServiceContainerNameMap[TypeServiceLifecycle] = 'ksan-lifecycle' -ServiceTypeDockerServiceContainerNameMap[TypeServiceRecovery] = 'ksan-recovery' +ServiceTypeDockerServiceContainerNameMap[TypeServiceOSD] = KsanOSDContainerName +ServiceTypeDockerServiceContainerNameMap[TypeServiceGW] = KsanGWContainerName +ServiceTypeDockerServiceContainerNameMap[TypeServiceLifecycle] = KsanLifecycleManagerContainerName +ServiceTypeDockerServiceContainerNameMap[TypeServiceRecovery] = KsanRecoveryName +ServiceTypeDockerServiceContainerNameMap[TypeServiceReplication] = KsanReplicationMangerContainerName +ServiceTypeDockerServiceContainerNameMap[TypeServiceLogManager] = KsanLogManagerContainerName -### service type converion ### +### service type conversion ### ServiceTypeConversion = dict() -ServiceTypeConversion['ksanosd'] = TypeServiceOSD -ServiceTypeConversion['ksangw'] = TypeServiceGW +ServiceTypeConversion[KsanOSDName.lower()] = TypeServiceOSD +ServiceTypeConversion[KsanGWName.lower()] = TypeServiceGW #ServiceTypeConversion['mongodb'] = TypeServiceMongoDB #ServiceTypeConversion['mariadb'] = TypeServiceMariaDB #ServiceTypeConversion['ksanmonitor'] = TypeServiceMonitor -ServiceTypeConversion['ksanagent'] = TypeServiceAgent -ServiceTypeConversion['ksanlifecycle'] = TypeServiceLifecycle -ServiceTypeConversion['ksanrecovery'] = TypeServiceRecovery +ServiceTypeConversion[KsanAgentName.lower()] = TypeServiceAgent +ServiceTypeConversion[KsanLifecycleManagerName.lower()] = TypeServiceLifecycle +#ServiceTypeConversion[KsanRecoveryName.lower()] = TypeServiceRecovery +ServiceTypeConversion[KsanReplicationManagerName.lower()] = TypeServiceReplication +ServiceTypeConversion[KsanLogManagerName.lower()] = TypeServiceLogManager #ServiceTypeConversion['rabbitmq'] = TypeServiceRabbitMq #ServiceTypeConversion['haproxy'] = TypeServiceHaproxy + +##### ksanAgent.conf Key Define ##### +KeyCommonSection = 'mgs' +KeyPortalHost = 'PortalHost' +KeyPortalPort = 'PortalPort' +KeyPortalApiKey = 'PortalApiKey' +KeyMQHost = 'MQHost' +KeyMQPort = 'MQPort' +KeyMQUser = 'MQUser' +KeyMQPassword = 'MQPassword' +KeyServerId = 'ServerId' +KeyManagementNetDev = 'ManagementNetDev' +KeyDefaultNetworkId = 'DefaultNetworkId' + +KeyMonitorSection = 'monitor' +KeyServerMonitorInterval = 'ServerMonitorInterval' +KeyNetworkMonitorInterval = 'NetworkMonitorInterval' +KeyDiskMonitorInterval = 'DiskMonitorInterval' +KeyServiceMonitorInterval = 'ServiceMonitorInterval' + +##### ksanAgent.conf Interval Define ##### +DiskMonitorInterval = 10 +ServerMonitorInterval = 10 +ServiceMonitorInterval = 10 +NetworkMonitorInterval = 10 +IntervalShort = 1 +IntervalMiddle = 5 +IntervalLong = 10 + + + +##### Utililty Variable Define ##### + +MoreDetailInfo = 'MoreDetail' +DetailInfo = 'Detail' +SimpleInfo = 'Simple' + + +##### Path Define ##### +KsanEtcPath = '/usr/local/ksan/etc' +KsanBinPath = '/usr/local/ksan/bin' +KsanSbinPath = '/usr/local/ksan/sbin' +KsanSslPath = '/usr/local/ksan/ssl' +KsanUtilDirPath = '/usr/local/ksan/bin/util' + +MonServicedConfPath = '/usr/local/ksan/etc/ksanAgent.conf' +DiskPoolXmlPath = '/usr/local/ksan/etc/diskpools.xml' +ServicePoolXmlPath = '/var/log/ksan/agent/servicepools_dump.xml' +OsdXmlFilePath = '/usr/local/ksan/etc/ksan-osd-log.xml' +GwXmlFilePath = '/usr/local/ksan/etc/ksan-gw-log.xml' +KsanMongDbManagerBinPath = '/usr/local/ksan/bin/ksanMongoDBManager' + + +KsanUtilPath = '%s/ksan' % KsanBinPath +KsanAgentPath = '%s/%s' % (KsanSbinPath, KsanAgentBinaryName) +KsanGwPath = '%s/%s' % (KsanSbinPath, KsanGwBinaryName) +KsanOsdPath = '%s/%s' % (KsanSbinPath, KsanOsdBinaryName) +KsanRecoveryPath = '%s/%s' % (KsanSbinPath, KsanRecoveryBinaryName) +KsanServerRegisterPath = '%s/%s' % (KsanUtilDirPath, KsanServerRegister) + + +### Service Action Retry Count ### +ServiceContolRetryCount = 3 + + + +##### Header of Response Body Class for deserialization ##### +#ResponseItemsHeaderModule = 'const.http.ResponseItemsHeader' +#ResponseHeaderModule = 'const.http.ResponseHeader' +#ResponseHeaderWithDataModule = 'const.http.ResponseHeaderWithData' + +ServerItemsModule = 'const.server.ServerItems' +ServerItemsDetailModule = 'const.server.ServerItemsDetail' +ServerUsageItemsModule = 'const.server.ServerUsageItems' +ServerStateItemsModule = 'const.server.ServerStateItems' + + +ServiceItemsDetailModule = 'const.service.ServiceItemsDetail' +ServiceDetailModule = 'const.service.ServiceDetail' +ServiceControlModule = 'const.service.ServiceControl' +ServiceGroupItemsModule = 'const.service.ServiceGroupItems' +ServiceGroupDetailModule = 'const.service.ServiceGroupDetail' +ServiceConfigModule = 'const.service.ServiceConfigItems' + +UserObjectModule = 'const.user.UserObject' +S3UserObjectModule = 'const.user.S3UserObject' + + +NetworkInterfaceItemsModule = 'const.network.NetworkInterfaceItems' +VlanNetworkInterfaceItemsModule = 'const.network.VlanNetworkInterfaceItems' + + +DiskItemsDetailModule = 'const.disk.DiskItemsDetail' +DiskDetailModule = 'const.disk.DiskDetail' +AllDiskItemsDetailModule = 'const.disk.AllDiskItemsDetail' +DiskPoolItemsModule = 'const.disk.DiskPoolItems' +DiskPoolDetailModule = 'const.disk.DiskPoolDetail' + +OnePlusOne = 'OnePlusOne' +OnePlusZero = 'OnePlusZero' +OnePlusTwo = 'OnePlusTwo' +ErasureCode = 'ErasureCode' + +Parsing = dict() + +Parsing['Server'] = ServerItemsModule +Parsing['NetworkInterfaces'] = NetworkInterfaceItemsModule +Parsing['NetworkInterfaceVlans'] = VlanNetworkInterfaceItemsModule +Parsing['Vlans'] = VlanNetworkInterfaceItemsModule +Parsing['Disks'] = DiskItemsDetailModule +Parsing['DiskPool'] = DiskPoolItemsModule +Parsing['Services'] = ServiceItemsDetailModule +Parsing['ServiceGroup'] = ServiceGroupItemsModule +Parsing['User'] = UserObjectModule + + + +''' +#### USER #### +''' +ValidStorageClassList = ['standard_ia', 'onezone_ia', 'intelligent_tiering', 'glacier', 'reduced_redundancy', 'deep_archive', 'outposts', 'glacier_ir'] + class AgentConf(BaseModel): LocalIp: str PortalHost: str @@ -358,3 +393,150 @@ class AgentConf(BaseModel): MQPassword: str MQUser: str PortalApiKey: str + + + + +""" +######### mq info define ######### +""" +MqVirtualHost = '/' +MqUser = 'ksanmq' +MqPassword = 'YOUR_MQ_PASSWORD' + +DiskStart = 'Good' +DiskStop = 'Stop' +MqDiskQueueName = 'disk' +MqDiskQueueExchangeName = 'disk' +MqDiskQueueRoutingKey = "*.services.disks.control" + +## server routing key +RoutKeyServerAdd = '*.servers.added' +RoutKeyServerAddFinder = re.compile('.servers.added') +RoutKeyServerDel = '*.servers.removed' +RoutKeyServerDelFinder = re.compile('.servers.removed') +RoutKeyServerUpdate = '*.servers.updated' +RoutKeyServerUpdateFinder = re.compile('.servers.updated') +RoutKeyServerState = '*.servers.state' +RoutKeyServerUsage = '*.servers.usage' + +## network routing key +RoutKeyNetwork = '.servers.interfaces.' +RoutKeyNetworkLinkState = '*.servers.interfaces.linkstate' +RoutKeyNetworkUsage = '*.servers.interfaces.usage' +RoutKeyNetworkVlanUsage = '*.servers.interfaces.vlans.usage' + +RoutKeyNetworkRpcFinder = re.compile('.servers.[\d\w-]+.interfaces.') +RoutKeyNetworkAddFinder = re.compile('.servers.[\d\w-]+.interfaces.add') +RoutKeyNetworkAddedFinder = re.compile('.servers.interfaces.added') +RoutKeyNetworkUpdateFinder = re.compile('.servers.[\d\w-]+.interfaces.update') + +## disk routing key +RoutKeyDisk = '.servers.disks.' +RoutKeyDiskAdded = '.servers.disks.added' +RoutKeyDiskDel = '.servers.disks.removed' +RoutKeyDiskState = '.servers.disks.state' +RoutKeyDiskHaAction = '.servers.disks.haaction' +RoutKeyDiskUsage = '.servers.disks.usage' +RoutKeyDiskUpdated = '.servers.disks.updated' +RoutKeyDiskGetMode = '.servers.disks.rwmode' +RoutKeyDiskSetMode = '.servers.disks.rwmode.update' +RoutKeyDiskStartStop = '.servers.disks.control' +## rpc +RoutKeyDiskRpcFinder = re.compile('.servers.[\w\d-]+.disks') +RoutKeyDiskCheckMountFinder = re.compile('.servers.[\w\d-]+.disks.check_mount') +RoutKeyDiskWirteDiskIdFinder = re.compile('.servers.[\w\d-]+.disks.write_disk_id') + +## disk pool routing key +RoutKeyDiskPool = 'servers.diskpools.' +RoutKeyDiskPoolAdd = 'servers.diskpools.added' +RoutKeyDiskPoolDel = 'servers.diskpools.removed' +RoutKeyDiskPoolUpdate = 'servers.diskpools.updated' + +## service routing key +RoutKeyService = '.services.' +RoutKeyServiceRpcFinder = re.compile('.services.[\d\w-]+.') +RoutKeyServiceState = '.services.state' +RoutKeyServiceHaAction = '.services.haaction' +RoutKeyServiceUsage = '.services.usage' +RoutKeyServiceControlFinder = re.compile('.services.[\d\w-]+.control') +RoutKeyServiceOsdConfLoadFinder = re.compile('.services.[\d\w-]+.config.osd.load') +RoutKeyServiceOsdConfSaveFinder = re.compile('.services.[\d\w-]+.config.osd.save') +RoutKeyServiceGwConfLoadFinder = re.compile('.services.[\d\w-]+.config.gw.load') +RoutKeyServiceGwConfSaveFinder = re.compile('.services.[\d\w-]+.config.gw.save') + +""" +EdgeRoutingKeyList = [ "*.servers.updated", "*.servers.removed", "*.servers.stat", "*.servers.usage", + "*.servers.interfaces.added", "*.servers.interfaces.updated", "*.servers.interfaces.removed", + "*.servers.interfaces.linkstate", "*.servers.interfaces.usage", "*.servers.interfaces.vlans.added", + "*.servers.interfaces.vlans.updated", "*.servers.interfaces.vlans.removed", + "*.servers.disks.added", "*.servers.disks.updated", "*.servers.disks.removed", "*.servers.disks.state", + "*.servers.disks.size", "*.servers.disks.rwmode", "*.servers.diskpools.added", "*.servers.diskpools.updated", + "*.servers.diskpools.removed", + "*.services.state", "*.services.stat", "*.services.haaction", "*.services.usage"] +""" + + +EdgeRoutingKeyList = [ "*.servers.updated", "*.servers.removed", "*.servers.added", + "*.servers.interfaces.added", "*.servers.interfaces.updated", "*.servers.interfaces.removed", + "*.servers.interfaces.vlans.added", + "*.servers.interfaces.vlans.updated", "*.servers.interfaces.vlans.removed", + "*.servers.disks.added", "*.servers.disks.updated", "*.servers.disks.removed", + "*.servers.disks.rwmode", "*.servers.diskpools.added", "*.servers.diskpools.updated", + "*.servers.diskpools.removed", "*.services.added", "*.services.updated", "*.services.removed"] + +MonRoutingKeyList = ["*.servers.updated", "*.servers.removed", "*.servers.interfaces.added", "*.servers.added", + "*.servers.interfaces.updated", "*.servers.interfaces.removed", "*.servers.interfaces.vlans.added" + , "*.servers.interfaces.vlans.updated", "*.servers.interfaces.vlans.removed", + "*.servers.disks.added", "*.servers.disks.updated", "*.servers.disks.removed", + "*.servers.disks.rwmode", "*.servers.diskpools.added", "*.servers.diskpools.updated", + "*.servers.diskpools.removed", "*.services.added", "*.services.updated"] + +## Exchange Name +ExchangeName = 'ksan.system' + + +##### return code & messages define ##### +RetKeyResult = 'Result' +RetKeyCode = 'Code' +RetKeyMessage = 'Message' + + + + +ResOk = 0 +ResNotFoundCode = 2 +ResNotFoundMsg = 'Not found ' +ResInvalidCode = 22 +ResInvalidMsg = 'Invalid Error ' +ResConnectionErrorCode = 111 +ResConnectionErrorMsg ='Connection Error ' +ResTimeErrorCode = 11 +ResTimeErrorCodeMsg = 'Timeout Error ' +ResDuplicateCode = 17 +ResDuplicateMsg = 'Duplicated' +ResDiskAlreadyExists = 'Disk Id already exists. Initialize first' + +ResEtcErrorCode = 1 +ResEtcErrorMsg = 'Other Error ' +ResFailToGetVlainId = 'Fail to get Vlan Id' + +ResultFail = 1 # fail to get data + +ResultSuccess = 'Success' +ResultWarning = 'Warning' +ResultError = 'Error' +CodeSuccess = 'EC000' +MessageNull = '' + +CodeFailServerInit = 'EC001' +MessageFailServerInit = 'Fail to init Server' + + +CodeDuplicated = 'EC036' +ResutlNotFound = 'EC014' + +ErrMsgServerNameMissing = 'ServerName is required' +ErrMsgDiskNameMissing = 'DiskName is required' +ErrMsgDiskpoolNameMissing = 'DiskpoolName is required' +ErrMsgUserNameMissing = 'UserName is required' diff --git a/core/common/const/disk.py b/core/common/const/disk.py index 87430547..5ee8b824 100644 --- a/core/common/const/disk.py +++ b/core/common/const/disk.py @@ -9,11 +9,6 @@ * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. """ -DiskItemsDetailModule = 'const.disk.DiskItemsDetail' -DiskDetailModule = 'const.disk.DiskDetail' -AllDiskItemsDetailModule = 'const.disk.AllDiskItemsDetail' -DiskPoolItemsModule = 'const.disk.DiskPoolItems' -DiskPoolDetailModule = 'const.disk.DiskPoolDetail' class DiskItemsDetail: def __init__(self): @@ -117,6 +112,7 @@ def Set(self, Id, ServerId, State, TotalInode, ReservedInode, class AddDiskObject: def __init__(self): + self.ServerId = '' self.DiskPoolId = '' self.Name = '' self.Path = '' @@ -129,8 +125,9 @@ def __init__(self): self.UsedSize = 0 self.Rwmode = 'ReadOnly' - def Set(self, Name, Path, State, TotalInode, ReservedInode, UsedInode, TotalSize, ReservedSize, UsedSize, RwMode, DiskPoolId=''): + def Set(self, ServerId, Name, Path, State, TotalInode, ReservedInode, UsedInode, TotalSize, ReservedSize, UsedSize, RwMode, DiskPoolId=''): + self.ServerId = ServerId self.DiskPoolId = DiskPoolId self.Name = Name self.Path = Path @@ -200,6 +197,10 @@ def __init__(self): self.Id = None self.Name = None self.Description = None + self.DiskPoolType = None + self.ReplicationType = None + self.EC = None + self.Disks = None self.ModDate = None self.ModId = None self.ModName = None diff --git a/core/common/const/http.py b/core/common/const/http.py deleted file mode 100644 index 50efc47b..00000000 --- a/core/common/const/http.py +++ /dev/null @@ -1,170 +0,0 @@ -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import sys -import json -from const.server import ServerItemsModule -from const.network import NetworkInterfaceItemsModule, VlanNetworkInterfaceItemsModule -from const.disk import DiskItemsDetailModule, DiskPoolItemsModule -from const.service import ServiceGroupItemsModule, ServiceItemsDetailModule -from const.user import UserObjectModule -from common.log import catch_exceptions - -##### Header of Response Body Class for deserialization ##### -ResponseItemsHeaderModule = 'const.http.ResponseItemsHeader' -ResponseHeaderModule = 'const.http.ResponseHeader' -ResponseHeaderWithDataModule = 'const.http.ResponseHeaderWithData' - -Parsing = dict() - -Parsing['Server'] = ServerItemsModule -Parsing['NetworkInterfaces'] = NetworkInterfaceItemsModule -Parsing['NetworkInterfaceVlans'] = VlanNetworkInterfaceItemsModule -Parsing['Vlans'] = VlanNetworkInterfaceItemsModule -Parsing['Disks'] = DiskItemsDetailModule -Parsing['DiskPool'] = DiskPoolItemsModule -Parsing['Services'] = ServiceItemsDetailModule -Parsing['ServiceGroup'] = ServiceGroupItemsModule -Parsing['User'] = UserObjectModule - -@catch_exceptions() -class GetApiResult: - def __init__(self, Header, ItemHeader, Items): - self.Header = Header - self.ItemHeader = ItemHeader - self.Items = Items - - -class ResponseHeader(object): - """ - Parsing Response without "Data" value is single value like True/False - """ - def __init__(self): - self.IsNeedLogin = None - self.AccessDenied = None - self.Result = None - self.Code = None - self.Message = None - - def Set(self, IsNeedLogin, AccessDenied, Result, Code, Message): - self.IsNeedLogin = IsNeedLogin - self.AccessDenied = AccessDenied - self.Result = Result - self.Code = Code - self.Message = Message - -class ResponseHeaderWithData(object): - """ - Parsing Response with "Data" value is multi value like dict or list - """ - def __init__(self): - self.IsNeedLogin = None - self.AccessDenied = None - self.Result = None - self.Code = None - self.Message = None - self.Data = None - - def Set(self, IsNeedLogin, AccessDenied, Result, Code, Message, Data): - self.IsNeedLogin = IsNeedLogin - self.AccessDenied = AccessDenied - self.Result = Result - self.Code = Code - self.Message = Message - self.Data = Data - - -class ResponseItemsHeader(object): - def __init__(self): - self.TotalCount = '' - self.Skips = '' - self.PageNo = '' - self.CountPerPage = '' - self.PagePerSection = '' - self.TotalPage = '' - self.StartPageNo = '' - self.EndPageNo = '' - self.PageNos = '' - self.HavePreviousPage = '' - self.HaveNextPage = '' - self.HavePreviousPageSection = '' - self.HaveNextPageSection = '' - self.Items = '' - - - - - -@catch_exceptions() -class ResPonseHeader(object): - def __init__(self, dic, logger=None): - try: - self.IsNeedLogin = dic['IsNeedLogin'] - self.AccessDenied = dic['AccessDenied'] - self.Result = dic['Result'] - self.Code = dic['Code'] - self.Message = dic['Message'] - if 'Data' in dic: - self.Data = dic['Data'] - except KeyError as err: - if logger is not None: - logger.error(err, sys.exc_info()[2].tb_lineno) - else: - print(err, sys.exc_info()[0].tb_lineno) - - -##### Response Body Decoder Class ##### -class ResPonseHeaderDecoder(json.JSONDecoder): - def __init__(self, logger=None, *args, **kwargs): - # json.JSONDecoder.__init__(self, object_hook=self.object_hook, *args, **kwargs) - self.logger=logger - json.JSONDecoder.__init__(self, object_hook=self.object_hook, *args, **kwargs) - - def object_hook(self, dct): - try: - print(">>>>", dct) - if 'IsNeedLogin' in dct: - obj = ResPonseHeader(dct) - if 'Data' in dct: - print(dct['Data']) - return obj - except KeyError as err: - if self.logger is not None: - self.logger.error(err, sys.exc_info()[2].tb_lineno) - else: - print(err, sys.exc_info()[0].tb_lineno) - - -##### Items Header of Response Body Class ##### -class ResPonseItemsHeader(object): - def __init__(self, dic, logger=None): - try: - self.TotalCount = dic['TotalCount'] - self.Skips = dic['Skips'] - self.PageNo = dic['PageNo'] - self.CountPerPage = dic['CountPerPage'] - self.PagePerSection = dic['PagePerSection'] - self.TotalPage = dic['TotalPage'] - self.StartPageNo = dic['StartPageNo'] - self.EndPageNo = dic['EndPageNo'] - self.PageNos = dic['PageNos'] - self.HavePreviousPage = dic['HavePreviousPage'] - self.HaveNextPage = dic['HaveNextPage'] - self.HavePreviousPageSection = dic['HavePreviousPageSection'] - self.HaveNextPageSection = dic['HaveNextPageSection'] - self.Items = dic['Items'] - except KeyError as err: - if logger is not None: - logger.error(err, sys.exc_info()[2].tb_lineno) - else: - print(err, sys.exc_info()[0].tb_lineno) - - diff --git a/core/common/const/mq.py b/core/common/const/mq.py index 2bbbc5ac..5a13d0f7 100644 --- a/core/common/const/mq.py +++ b/core/common/const/mq.py @@ -9,102 +9,80 @@ * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. """ import re +from enum import Enum +from const.common import * +import json +import pika -""" -######### mq info define ######### -""" -MqVirtualHost = '/' -MqUser = 'ksanmq' -MqPassword = 'YOUR_MQ_PASSWORD' - -DiskStart = 'Good' -DiskStop = 'Stop' -MqDiskQueueName = 'disk' -MqDiskQueueExchangeName = 'disk' -MqDiskQueueRoutingKey = "*.services.disks.control" - -## server routing key -RoutKeyServerAdd = '*.servers.added' -RoutKeyServerAddFinder = re.compile('.servers.added') -RoutKeyServerDel = '*.servers.removed' -RoutKeyServerDelFinder = re.compile('.servers.removed') -RoutKeyServerUpdate = '*.servers.updated' -RoutKeyServerUpdateFinder = re.compile('.servers.updated') -RoutKeyServerState = '*.servers.state' -RoutKeyServerUsage = '*.servers.usage' - -## network routing key -RoutKeyNetwork = '.servers.interfaces.' -RoutKeyNetworkLinkState = '*.servers.interfaces.linkstate' -RoutKeyNetworkUsage = '*.servers.interfaces.usage' -RoutKeyNetworkVlanUsage = '*.servers.interfaces.vlans.usage' - -RoutKeyNetworkRpcFinder = re.compile('.servers.[\d\w-]+.interfaces.') -RoutKeyNetworkAddFinder = re.compile('.servers.[\d\w-]+.interfaces.add') -RoutKeyNetworkAddedFinder = re.compile('.servers.interfaces.added') -RoutKeyNetworkUpdateFinder = re.compile('.servers.[\d\w-]+.interfaces.update') - -## disk routing key -RoutKeyDisk = '.servers.disks.' -RoutKeyDiskAdded = '.servers.disks.added' -RoutKeyDiskDel = '.servers.disks.removed' -RoutKeyDiskState = '.servers.disks.state' -RoutKeyDiskHaAction = '.servers.disks.haaction' -RoutKeyDiskUsage = '.servers.disks.usage' -RoutKeyDiskUpdated = '.servers.disks.updated' -RoutKeyDiskGetMode = '.servers.disks.rwmode' -RoutKeyDiskSetMode = '.servers.disks.rwmode.update' -RoutKeyDiskStartStop = '.servers.disks.control' -## rpc -RoutKeyDiskRpcFinder = re.compile('.servers.[\w\d-]+.disks') -RoutKeyDiskCheckMountFinder = re.compile('.servers.[\w\d-]+.disks.check_mount') -RoutKeyDiskWirteDiskIdFinder = re.compile('.servers.[\w\d-]+.disks.write_disk_id') - -## disk pool routing key -RoutKeyDiskPool = 'servers.diskpools.' -RoutKeyDiskPoolAdd = 'servers.diskpools.added' -RoutKeyDiskPoolDel = 'servers.diskpools.removed' -RoutKeyDiskPoolUpdate = 'servers.diskpools.updated' - -## service routing key -RoutKeyService = '.services.' -RoutKeyServiceRpcFinder = re.compile('.services.[\d\w-]+.') -RoutKeyServiceState = '.services.state' -RoutKeyServiceHaAction = '.services.haaction' -RoutKeyServiceUsage = '.services.usage' -RoutKeyServiceControlFinder = re.compile('.services.[\d\w-]+.control') -RoutKeyServiceOsdConfLoadFinder = re.compile('.services.[\d\w-]+.config.osd.load') -RoutKeyServiceOsdConfSaveFinder = re.compile('.services.[\d\w-]+.config.osd.save') -RoutKeyServiceGwConfLoadFinder = re.compile('.services.[\d\w-]+.config.gw.load') -RoutKeyServiceGwConfSaveFinder = re.compile('.services.[\d\w-]+.config.gw.save') +class EnumResponseResult(Enum): + Error = -1 + Warning = 0 + Success = 1 -""" -EdgeRoutingKeyList = [ "*.servers.updated", "*.servers.removed", "*.servers.stat", "*.servers.usage", - "*.servers.interfaces.added", "*.servers.interfaces.updated", "*.servers.interfaces.removed", - "*.servers.interfaces.linkstate", "*.servers.interfaces.usage", "*.servers.interfaces.vlans.added", - "*.servers.interfaces.vlans.updated", "*.servers.interfaces.vlans.removed", - "*.servers.disks.added", "*.servers.disks.updated", "*.servers.disks.removed", "*.servers.disks.state", - "*.servers.disks.size", "*.servers.disks.rwmode", "*.servers.diskpools.added", "*.servers.diskpools.updated", - "*.servers.diskpools.removed", - "*.services.state", "*.services.stat", "*.services.haaction", "*.services.usage"] -""" -EdgeRoutingKeyList = [ "*.servers.updated", "*.servers.removed", "*.servers.added", - "*.servers.interfaces.added", "*.servers.interfaces.updated", "*.servers.interfaces.removed", - "*.servers.interfaces.vlans.added", - "*.servers.interfaces.vlans.updated", "*.servers.interfaces.vlans.removed", - "*.servers.disks.added", "*.servers.disks.updated", "*.servers.disks.removed", - "*.servers.disks.rwmode", "*.servers.diskpools.added", "*.servers.diskpools.updated", - "*.servers.diskpools.removed", "*.services.added", "*.services.updated", "*.services.removed"] +from typing import TypeVar, Generic, Type + +T = TypeVar('T') + +class ResponseData: + def __init__(self, result: EnumResponseResult = EnumResponseResult.Error, code: str = "", messsage: str = "") -> None: + self.Result = result + self.Code = code + self.Message = messsage + + def load(self, data: dict): + self.Result = data[RetKeyResult] + self.Code = data[RetKeyCode] + self.Message = data[RetKeyMessage] + + + +def MqReturn(Result, Code=0, Messages='', Data=None): + Ret = dict() + if Result is True or Result == ResultSuccess: + Ret['Result'] = 'Success' + else: + Ret['Result'] = 'Error' + Ret['Code'] = Code + + Ret['Message'] = Messages + if Data: + Ret['Data'] = Data + #Ret['Data'] = json.dumps(Ret) + return json.dumps(Ret) + +def RemoveQueue(QueueHost, QueueName): + connection = pika.BlockingConnection(pika.ConnectionParameters(QueueHost)) + channel = connection.channel() + + channel.queue_delete(queue=QueueName) + connection.close() + + + + +class ResponseDataWithData(Generic[T]): + def __init__(self, result: EnumResponseResult = EnumResponseResult.Error, code: str = "", message: str = "", value: T = None) -> None: + super().__init__() + self.Result = result + self.Code = code + self.Message = message + self.Data = value + + def load(self, data: dict): + self.__dict__ = data.copy() + -MonRoutingKeyList = ["*.servers.updated", "*.servers.removed", "*.servers.interfaces.added", "*.servers.added", - "*.servers.interfaces.updated", "*.servers.interfaces.removed", "*.servers.interfaces.vlans.added" - , "*.servers.interfaces.vlans.updated", "*.servers.interfaces.vlans.removed", - "*.servers.disks.added", "*.servers.disks.updated", "*.servers.disks.removed", - "*.servers.disks.rwmode", "*.servers.diskpools.added", "*.servers.diskpools.updated", - "*.servers.diskpools.removed", "*.services.added", "*.services.updated"] +class ResponseMqData: + def __init__(self, result: EnumResponseResult = EnumResponseResult.Error, code: str = "", messsage: str = "") -> None: + self.Result = result + self.Code = code + self.Message = messsage + self.IsProcessed = False -## Exchange Name -ExchangeName = 'ksan.system' \ No newline at end of file + def load(self, data: dict): + self.Result = data[RetKeyResult] + self.Code = data[RetKeyCode] + self.Message = data[RetKeyMessage] diff --git a/core/common/const/network.py b/core/common/const/network.py index cd5d57ca..e25955d9 100644 --- a/core/common/const/network.py +++ b/core/common/const/network.py @@ -8,10 +8,8 @@ * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. """ -from common.log import catch_exceptions - -NetworkInterfaceItemsModule = 'const.network.NetworkInterfaceItems' -VlanNetworkInterfaceItemsModule = 'const.network.VlanNetworkInterfaceItems' +#from common.log import catch_exceptions +#from const.common import * class NetworkInterfaceItems: @@ -37,7 +35,6 @@ def __init__(self): self.ModName = '' -@catch_exceptions() class RequestNetworkInterfaceItems(object): """ request to register network info to ifsportalsvr @@ -69,7 +66,6 @@ def Set(self, Nic): self.Dns2 = Nic['Dns2'] -@catch_exceptions() class VlanNetworkInterfaceItems: def __init__(self): self.Id = None @@ -90,7 +86,6 @@ def __init__(self): self.RegName = None -@catch_exceptions() class RequestNetworkInterfaceCheck(object): """ request to register network info to ifsportalsvr @@ -99,7 +94,6 @@ def __init__(self, NicName, logger=None): self.Name = NicName -@catch_exceptions() class NetworkInterfaceLinkStateItems(object): """ request to update network interface status to ifsportalsvr @@ -111,7 +105,6 @@ def __init__(self, InterfaceId, ServerId, LinkState): self.LinkState = LinkState -@catch_exceptions() class RequestNetworkInterfaceStat(object): """ request to update network interface stat to ifsportalsvr @@ -138,7 +131,6 @@ def __init__(self, Tag=None, IpAddress=None, SubnetMask=None, Gateway=None): self.SubnetMask = SubnetMask self.Gateway = Gateway -@catch_exceptions() class ResPonseVlanNetworkInterfaceItems(object): def __init__(self, dic): self.Id = dic["Id"] @@ -155,7 +147,6 @@ def __init__(self, dic): self.RegName = dic["RegName"] -@catch_exceptions() class RequestVlanNetworkInterfaceCheck(object): """ request to register network info to ifsportalsvr diff --git a/core/common/const/server.py b/core/common/const/server.py index c5357297..0b877bd6 100644 --- a/core/common/const/server.py +++ b/core/common/const/server.py @@ -8,16 +8,19 @@ * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. """ -import sys +import os, sys +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from typing import TYPE_CHECKING + import socket, platform -import psutil -from common.log import catch_exceptions -from const.common import ServerStateOffline +#from common.log import catch_exceptions +#from const.common import ServerStateOffline +from const.common import * +#from const.common import ServerStateOffline +#from common.log import catch_exceptions + -ServerItemsModule = 'const.server.ServerItems' -ServerItemsDetailModule = 'const.server.ServerItemsDetail' -ServerUsageItemsModule = 'const.server.ServerUsageItems' -ServerStateItemsModule = 'const.server.ServerStateItems' class ServerItems(object): @@ -39,7 +42,6 @@ def __init__(self, Description='', State=ServerStateOffline): self.ModId = '' self.ModName = '' - @catch_exceptions() def Set(self, dic): self.Id = dic["Id"] self.Name = dic["Name"] diff --git a/core/common/const/service.py b/core/common/const/service.py index 6c2304d3..c8a4e10a 100644 --- a/core/common/const/service.py +++ b/core/common/const/service.py @@ -10,13 +10,6 @@ """ import re -ServiceItemsDetailModule = 'const.service.ServiceItemsDetail' -ServiceDetailModule = 'const.service.ServiceDetail' -ServiceControlModule = 'const.service.ServiceControl' -ServiceGroupItemsModule = 'const.service.ServiceGroupItems' -ServiceGroupDetailModule = 'const.service.ServiceGroupDetail' -ServiceConfigModule = 'const.service.ServiceConfigItems' - SystemdServicePidFinder = re.compile('Main PID: ([\d]+)') class AddServiceInfoObject: @@ -134,6 +127,7 @@ def __init__(self): self.MemoryTotal = None self.MemoryUsed = None self.ThreadCount = None + self.Server = None self.ModDate = None self.ModId = None self.ModName = None diff --git a/core/common/const/user.py b/core/common/const/user.py index b337385b..4f1b6c92 100644 --- a/core/common/const/user.py +++ b/core/common/const/user.py @@ -9,8 +9,7 @@ * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. """ -UserObjectModule = 'const.user.UserObject' -S3UserObjectModule = 'const.user.S3UserObject' + class AddUserObject: def __init__(self): diff --git a/core/common/disk/disk_manage.py b/core/common/disk/disk_api.py similarity index 60% rename from core/common/disk/disk_manage.py rename to core/common/disk/disk_api.py index 53d59b79..291dad66 100644 --- a/core/common/disk/disk_manage.py +++ b/core/common/disk/disk_api.py @@ -15,14 +15,14 @@ import time if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.shcommand import * -from common.utils import Byte2HumanValue, DisplayDiskState, DisplayDiskMode -from const.disk import AddDiskObject, UpdateDiskSizeObject, DiskDetailModule, AllDiskItemsDetailModule -from server.server_manage import * -from optparse import OptionParser -import re -import psutil -import jsonpickle +#from const.mq import * +from const.common import * +from common.utils import * +#from const.common import DiskDetailModule, AllDiskItemsDetailModule +from const.disk import AddDiskObject, UpdateDiskSizeObject +#from common.httpapi import RestApi +from common.base_utils import * +from portal_api.apis import * ParsingDiskInode = re.compile("([\d\w_\-/]+)[\s]+([\d]+)[\s]+([\d]+)[\s]+([\d]+)[\s]+([\d])\%") @@ -87,7 +87,7 @@ def AddDisk(Ip, Port, ApiKey, Path, DiskName, ServerId=None, ServerName=None, Di Url = '/api/v1/Disks/%s' % TargetServer disk = AddDiskObject() - disk.Set(DiskName, Path, DiskStop, 0, 0, 0, 0, 0, 0, DiskModeRw, DiskPoolId=DiskPoolId) + disk.Set(ServerId, DiskName, Path, DiskStop, 0, 0, 0, 0, 0, 0, DiskModeRw, DiskPoolId=DiskPoolId) body = jsonpickle.encode(disk, make_refs=False) Conn = RestApi(Ip, Port, Url, authkey=ApiKey, params=body, logger=logger) Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) @@ -129,6 +129,40 @@ def UpdateDiskInfo(Ip, Port, ApiKey, DiskId=None, DiskPoolId=None, Path=None, Na return Res, Errmsg, None +def UpdateDiskInfoDetail(PortalIp, PortalPort, PortalApiKey, DiskId, logger=None): + + # get server info + Res, Errmsg, DiskData, ServerId = GetDiskInfoWithId(PortalIp, PortalPort, PortalApiKey, DiskId=DiskId, logger=logger) + if Res == ResOk: + if DiskData is not None: + InputServerInfo = [{'key': 'ServerId', 'value': DiskData.ServerId, 'type': str, 'question': 'Insert new disk\'s server id'}, + {'key': 'DiskPoolId', 'value': DiskData.DiskPoolId, 'type': str, 'question': 'Insert new disk\'s diskpool id'}, + {'key': 'Name', 'value': DiskData.Name, 'type': str, 'question': 'Insert new disk name'}, + {'key': 'Path', 'value': DiskData.Path, 'type': str, 'question': 'Insert new disk path'} + ] + for info in InputServerInfo: + + QuestionString = info['question'] + ValueType = info['type'] + DefaultValue = info['value'] + DiskData.__dict__[info['key']] = get_input(QuestionString, ValueType, DefaultValue) + + Url = '/api/v1/Disks/%s' % DiskId + disk = AddDiskObject() + disk.Set(DiskData.__dict__['ServerId'], DiskData.Name, DiskData.Path, DiskData.State, DiskData.TotalInode, DiskData.ReservedInode, + DiskData.UsedInode, DiskData.TotalSize, DiskData.ReservedSize, DiskData.UsedSize, DiskData.RwMode, DiskPoolId=DiskData.DiskPoolId) + body = jsonpickle.encode(disk, make_refs=False) + Params = body + Conn = RestApi(PortalIp, PortalPort, Url, authkey=PortalApiKey, params=Params, logger=logger) + Ret, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResPonseHeader) + if Ret != ResOk: + return Ret, Errmsg + else: + return Data.Result, Data.Message + else: + return Res, Errmsg + + @catch_exceptions() def RemoveDiskInfo(ip, port, ApiKey, DiskId=None, Name=None, logger=None): # get network interface info @@ -325,59 +359,165 @@ def ShowDiskInfo(DiskList, ServerId=None, DiskId=None, Detail=None, Continue=Non :return: """ - if Detail in [DetailInfo, MoreDetailInfo]: - DiskTitleLine = '%s' % ('=' * 316) - DiskDataLine = '%s' % ('-' * 316) - title = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ('ServerName'.center(15), 'DiskName'.center(15), 'Path'.center(15), - 'TotalSize'.center(9), 'UsedSize'.center(8), 'FreeSize'.center(8), - 'TotalInode'.center(20), 'UsedInode'.center(20), 'ReservedInode'.center(20), 'Read'.center(6), 'Write'.center(6), - 'RwMode'.center(6), 'DiskId'.center(38), 'DiskPoolId'.center(38), 'State'.center(7)) # 'ModeDate'.center(20), 'ModId'.center(20), 'ModName'.center(20), 'Id'.center(30)) - elif Detail is SimpleInfo: - DiskTitleLine = '%s' % ('=' * 106) - DiskDataLine = '%s' % ('-' * 106) - title = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ('ServerName'.center(15), 'DiskName'.center(15), 'Path'.center(15), - 'TotalSize'.center(9), 'UsedSize'.center(8), 'FreeSize'.center(8), - 'Read'.center(6), 'Write'.center(6), - 'RwMode'.center(6), 'State'.center(7)) # 'ModeDate'.center(20), 'ModId'.center(20), 'ModName'.center(20), 'Id'.center(30)) + DiskList = DiskListOrdering(DiskList) + + if Detail == MoreDetailInfo: + DiskTitleLine = '%s' % ('=' * 228) + DiskDataLine = '%s' % ('-' * 228) + title = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ('ServerName'.ljust(15), 'DiskName'.ljust(15), 'Path'.ljust(15), + 'TotalSize'.ljust(9), 'UsedSize'.ljust(8), 'FreeSize'.ljust(8), 'Read'.ljust(8), 'Write'.ljust(8), + 'RwMode'.ljust(6), 'Status'.ljust(7), 'TotalInode'.ljust(20), 'UsedInode'.ljust(20), + 'ReservedInode'.ljust(20) , 'DiskPoolName'.ljust(15), 'DiskId'.ljust(38)) # 'ModeDate'.ljust(20), 'ModId'.ljust(20), 'ModName'.ljust(20), 'Id'.ljust(30)) + + elif Detail == DetailInfo: + DiskTitleLine = '%s' % ('=' * 189) + DiskDataLine = '%s' % ('-' * 189) + title = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ('ServerName'.ljust(15), 'DiskName'.ljust(15), 'Path'.ljust(15), + 'TotalSize'.ljust(9), 'UsedSize'.ljust(8), 'FreeSize'.ljust(8), 'Read'.ljust(8), 'Write'.ljust(8), + 'RwMode'.ljust(6), 'Status'.ljust(7), 'TotalInode'.ljust(20), + 'UsedInode'.ljust(20), 'ReservedInode'.ljust(20), 'DiskPoolName'.ljust(15)) # 'ModeDate'.ljust(20), 'ModId'.ljust(20), 'ModName'.ljust(20), 'Id'.ljust(30)) else: - DiskTitleLine = '%s' % ('=' * 64) - DiskDataLine = '%s' % ('-' * 64) - title = "|%s|%s|%s|%s|%s|" % ('ServerName'.center(15), 'DiskName'.center(15), 'Path'.center(15), 'RwMode'.center(6), 'State'.center(7)) + DiskTitleLine = '%s' % ('=' * 110) + DiskDataLine = '%s' % ('-' * 110) + title = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ('ServerName'.ljust(15), 'DiskName'.ljust(15), 'Path'.ljust(15), + 'TotalSize'.ljust(9), 'UsedSize'.ljust(8), 'FreeSize'.ljust(8), + 'Read'.ljust(8), 'Write'.ljust(8), + 'RwMode'.ljust(6), 'Status'.ljust(7)) # 'ModeDate'.ljust(20), 'ModId'.ljust(20), 'ModName'.ljust(20), 'Id'.ljust(30)) + #else: + # DiskTitleLine = '%s' % ('=' * 64) + # DiskDataLine = '%s' % ('-' * 64) + # title = "|%s|%s|%s|%s|%s|" % ('ServerName'.ljust(15), 'DiskName'.ljust(15), 'Path'.ljust(15), 'RwMode'.ljust(6), 'State'.ljust(7)) print(DiskTitleLine) print(title) print(DiskTitleLine) for disk in DiskList: - #if DiskId is not None and disk.Id != DiskId: # continue # svr = GetDataFromBody(disk.Server, ServerItemsModule) #if ServerId is not None and ServerId != svr.Id: # continue - if Detail in [DetailInfo, MoreDetailInfo]: - _dsp = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (disk.Server.Name.center(15), disk.Name.center(15), - '{:15.15}'.format(disk.Path.center(15)), - str(Byte2HumanValue(int(disk.TotalSize), 'TotalSize')).center(9), - str(Byte2HumanValue(int(disk.UsedSize), 'UsedSize')).center(8), str(Byte2HumanValue(int(disk.TotalSize - disk.UsedSize - disk.ReservedSize), 'FreeSize')).center(8), - str(int(disk.TotalInode)).center(20), str(int(disk.UsedInode)).center(20), str(int(disk.ReservedInode)).center(20), - str(Byte2HumanValue(disk.Read, 'DiskRW')).center(6), str(Byte2HumanValue(disk.Write, 'DiskRw')).center(6), - DisplayDiskMode(disk.RwMode).center(6),str(disk.Id).center(38) ,str(disk.DiskPoolId).center(38), DisplayDiskState(disk.State).center(7)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - elif Detail == SimpleInfo: - _dsp = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (disk.Server.Name.center(15), disk.Name.center(15), - '{:15.15}'.format(disk.Path.center(15)), - str(Byte2HumanValue(int(disk.TotalSize), 'TotalSize')).center(10), - str(Byte2HumanValue(int(disk.UsedSize), 'UsedSize')).center(10), str(Byte2HumanValue(int(disk.TotalSize - disk.UsedSize - disk.ReservedSize), 'FreeSize')).center(10), - str(Byte2HumanValue(disk.Read, 'DiskRw')).center(6), str(Byte2HumanValue(disk.Write, 'DiskRw')).center(6), - DisplayDiskMode(disk.RwMode).center(6), DisplayDiskState(disk.State).center(7)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - else: + if Detail == MoreDetailInfo: + _dsp = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (disk.Server.Name.ljust(15), disk.Name.ljust(15), + '{:15.15}'.format(disk.Path.ljust(15)), str(Byte2HumanValue(int(disk.TotalSize), 'TotalSize')).rjust(9), + str(Byte2HumanValue(int(disk.UsedSize), 'UsedSize')).rjust(8), + str(Byte2HumanValue(int(disk.TotalSize - disk.UsedSize - disk.ReservedSize), 'FreeSize')).rjust(8), + str(Byte2HumanValue(disk.Read, 'DiskRw')).rjust(8), str(Byte2HumanValue(disk.Write, 'DiskRw')).rjust(8), + DisplayDiskMode(disk.RwMode).ljust(6), DisplayDiskState(disk.State).ljust(7), + "{:,}".format(int(disk.TotalInode)).rjust(20), "{:,}".format(int(disk.UsedInode)).rjust(20), "{:,}".format(int(disk.ReservedInode)).rjust(20), + str(disk.DiskPoolName).ljust(15), str(disk.Id).ljust(38)) # svr.ModDate.rjust(20), svr.ModId.rjust(20), svr.ModName.rjust(20), svr.Id.rjust(30)) + elif Detail == DetailInfo: + _dsp = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (disk.Server.Name.ljust(15), disk.Name.ljust(15), + '{:15.15}'.format(disk.Path.ljust(15)), + str(Byte2HumanValue(int(disk.TotalSize), 'TotalSize')).rjust(9), + str(Byte2HumanValue(int(disk.UsedSize), 'UsedSize')).rjust(8), str(Byte2HumanValue(int(disk.TotalSize - disk.UsedSize - disk.ReservedSize), 'FreeSize')).rjust(8), + str(Byte2HumanValue(disk.Read, 'DiskRw')).rjust(8), str(Byte2HumanValue(disk.Write, 'DiskRw')).rjust(8), + DisplayDiskMode(disk.RwMode).ljust(6), + DisplayDiskState(disk.State).ljust(7),"{:,}".format(int(disk.TotalInode)).rjust(20), "{:,}".format(int(disk.UsedInode)).rjust(20), + "{:,}".format(int(disk.ReservedInode)).rjust(20), str(disk.DiskPoolName).ljust(15)) # svr.ModDate.rjust(20), svr.ModId.rjust(20), svr.ModName.rjust(20), svr.Id.rjust(30)) - _dsp = "|%s|%s|%s|%s|%s|" % (disk.Server.Name.center(15), disk.Name.center(15), - disk.Path.center(15), DisplayDiskMode(disk.RwMode).center(6), - DisplayDiskState(disk.State).center(7)) + else: + _dsp = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (disk.Server.Name.ljust(15), disk.Name.ljust(15), + '{:15.15}'.format(disk.Path.ljust(15)), + str(Byte2HumanValue(int(disk.TotalSize), 'TotalSize')).rjust(10), + str(Byte2HumanValue(int(disk.UsedSize), 'UsedSize')).rjust(10), str(Byte2HumanValue(int(disk.TotalSize - disk.UsedSize - disk.ReservedSize), 'FreeSize')).rjust(10), + str(Byte2HumanValue(disk.Read, 'DiskRw')).rjust(8), str(Byte2HumanValue(disk.Write, 'DiskRw')).rjust(8), + DisplayDiskMode(disk.RwMode).ljust(6), DisplayDiskState(disk.State).ljust(7)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) + #else: + + # _dsp = "|%s|%s|%s|%s|%s|" % (disk.Server.Name.ljust(15), disk.Name.ljust(15), + # disk.Path.ljust(15), DisplayDiskMode(disk.RwMode).ljust(6), + # DisplayDiskState(disk.State).ljust(7)) print(_dsp) print(DiskDataLine) +def DiskListOrdering(DiskList): + TotalNewDiskList = list() + ServerNameDict = dict() + for diskinfo in DiskList: + ServerName = diskinfo.Server.Name + if ServerName not in ServerNameDict: + ServerNameDict[ServerName] = list() + ServerNameDict[ServerName].append(diskinfo) + else: + ServerNameDict[ServerName].append(diskinfo) + + for servername in sorted(ServerNameDict.keys(), key=str.casefold): + disklist = ServerNameDict[servername] + DiskNameDict = dict() + for idx, disk in enumerate(disklist): + DiskNameDict[disk.Name] = disk + + newdisklist = list() + for diskname in sorted(DiskNameDict.keys(), key=str.casefold) : + disk = DiskNameDict[diskname] + newdisklist.append(disk) + + TotalNewDiskList += newdisklist + + return TotalNewDiskList + + + +def MqDiskHandler(RoutingKey, Body, Response, ServerId, GlobalFlag, logger): + logger.debug("%s %s" % (str(RoutingKey), str(Body))) + if RoutKeyDiskCheckMountFinder.search(RoutingKey): + ResponseReturn = MqReturn(ResultSuccess) + Body = Body.decode('utf-8') + Body = json.loads(Body) + body = DictToObject(Body) + if ServerId == body.ServerId: + ret = CheckDiskMount(body.Path) + if ret is False: + ResponseReturn = MqReturn(ret, Code=1, Messages='No such disk is found') + Response.IsProcessed = True + logger.debug(ResponseReturn) + return ResponseReturn + elif RoutKeyDiskWirteDiskIdFinder.search(RoutingKey): + ResponseReturn = MqReturn(ResultSuccess) + Body = Body.decode('utf-8') + Body = json.loads(Body) + body = DictToObject(Body) + if ServerId == body.ServerId: + ret, errmsg = WriteDiskId(body.Path, body.Id) + if ret is False: + ResponseReturn = MqReturn(ret, Code=1, Messages=errmsg) + Response.IsProcessed = True + logger.debug(ResponseReturn) + return ResponseReturn + elif RoutingKey.endswith((RoutKeyDiskAdded, RoutKeyDiskDel, RoutKeyDiskUpdated)): + Body = Body.decode('utf-8') + Body = json.loads(Body) + body = DictToObject(Body) + GlobalFlag['DiskUpdated'] = Updated + logger.debug("disk updated %s" % body.Id) + logging.log(logging.INFO, "Disk Info is Added") + ResponseReturn = MqReturn(ResultSuccess) + Response.IsProcessed = True + logger.debug(ResponseReturn) + return ResponseReturn + elif RoutingKey.endswith(RoutKeyDiskPoolUpdate) or \ + RoutingKey.endswith(RoutKeyDiskStartStop) or \ + RoutingKey.endswith(RoutKeyDiskState) or \ + RoutingKey.endswith(RoutKeyDiskDel): + logging.log(logging.INFO, "Disk Info is Updated") + GlobalFlag['DiskUpdated'] = Updated + GlobalFlag['DiskPoolUpdated'] = Updated + ResponseReturn = MqReturn(ResultSuccess) + Body = Body.decode('utf-8') + Body = json.loads(Body) + body = DictToObject(Body) + Response.IsProcessed = True + logger.debug(ResponseReturn) + #UpdateDiskPoolXml() + return ResponseReturn + else: + ResponseReturn = MqReturn(ResultSuccess) + Response.IsProcessed = True + return ResponseReturn + + def DiskUtilHandler(Conf, Action, Parser, logger): @@ -415,14 +555,8 @@ def DiskUtilHandler(Conf, Action, Parser, logger): if not options.DiskName: Parser.print_help() sys.exit(-1) - Res, Errmsg, Ret = UpdateDiskInfo(PortalIp, PortalPort, PortalApiKey, Path=options.Path, - DiskPoolId=options.DiskPoolId, Name=options.DiskName, - Description=options.Description, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - + Res, Errmsg = UpdateDiskInfoDetail(PortalIp, PortalPort, PortalApiKey, options.DiskName, logger=logger) + print(Res, Errmsg) elif Action.lower() == 'update_size': if not options.DiskName: @@ -481,3 +615,4 @@ def DiskUtilHandler(Conf, Action, Parser, logger): time.sleep(int(options.Continue)) else: Parser.print_help() + diff --git a/core/common/disk/disk_mq_handle.py b/core/common/disk/disk_mq_handle.py index 4efcb49c..c7bb51c5 100644 --- a/core/common/disk/disk_mq_handle.py +++ b/core/common/disk/disk_mq_handle.py @@ -12,18 +12,13 @@ # -*- coding: utf-8 -*- import os, sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from disk.disk_manage import * -from const.common import DiskPoolXmlPath +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) from const.disk import DiskDetailMqBroadcast -from common.init import GetConf, WaitAgentConfComplete, GetAgentConfig -from const.mq import MqVirtualHost, RoutKeyDiskUsage, ExchangeName, RoutKeyDiskCheckMountFinder, \ - RoutKeyDiskWirteDiskIdFinder, RoutKeyDiskAdded, RoutKeyDiskPoolUpdate, RoutKeyDiskStartStop, RoutKeyDiskState, \ - RoutKeyDiskDel import mqmanage.mq +from portal_api.apis import * import time import json -import logging import inspect import xml.etree.ElementTree as ET @@ -60,7 +55,7 @@ def ReportDiskIo(Conf, DiskStatInfo, GlobalFlag, logger): Conf = WaitAgentConfComplete(inspect.stack()[1][3], logger) conf = GetAgentConfig(Conf) MqDiskUpdated = mqmanage.mq.Mq(conf.MQHost, int(conf.MQPort), MqVirtualHost, conf.MQUser, conf.MQPassword, RoutKeyDiskUsage, ExchangeName, - QueueName='') + QueueName='', logger=logger) DiskMonitorInterval = int(conf.DiskMonitorInterval)/1000 Res, Errmsg, Ret, DiskList = GetDiskInfo(conf.PortalHost, int(conf.PortalPort), conf.PortalApiKey) @@ -109,9 +104,11 @@ def UpdateDiskPartitionInfo(DiskList): """ DiskStatInfo = list() for disk in DiskList: + DiskPath = '%s/' % disk.Path for part in psutil.disk_partitions(): TmpDiskInfo = dict() - if disk.Path == part.mountpoint: + PartitionMountPath = '%s/' % part.mountpoint + if DiskPath.startswith(PartitionMountPath): Device = re.sub('/dev/', '', part.device) TmpDiskInfo['Path'] = disk.Path TmpDiskInfo['Device'] = Device @@ -123,6 +120,13 @@ def UpdateDiskPartitionInfo(DiskList): TmpDiskInfo['ReadPerSec'] = 0 TmpDiskInfo['WritePerSec'] = 0 TmpDiskInfo['LapTime'] = time.time() + TmpDiskInfo['TotalInode'] = disk.TotalInode + TmpDiskInfo['UsedInode'] = disk.UsedInode + TmpDiskInfo['ReservedInode'] = disk.ReservedInode + TmpDiskInfo['TotalSize'] = disk.TotalSize + TmpDiskInfo['UsedSize'] = disk.UsedSize + TmpDiskInfo['ReservedSize'] = disk.ReservedSize + DiskStatInfo.append(TmpDiskInfo) return DiskStatInfo @@ -237,62 +241,6 @@ def DiskUsageMonitoring(Conf, DiskStatInfo, GlobalFlag, logger): time.sleep(IntervalMiddle) -def MqDiskHandler(RoutingKey, Body, Response, ServerId, GlobalFlag, logger): - logger.debug("%s %s" % (str(RoutingKey), str(Body))) - if RoutKeyDiskCheckMountFinder.search(RoutingKey): - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Body = Body.decode('utf-8') - Body = json.loads(Body) - body = DictToObject(Body) - if ServerId == body.ServerId: - ret = CheckDiskMount(body.Path) - if ret is False: - ResponseReturn = mqmanage.mq.MqReturn(ret, Code=1, Messages='No such disk is found') - Response.IsProcessed = True - logger.debug(ResponseReturn) - return ResponseReturn - elif RoutKeyDiskWirteDiskIdFinder.search(RoutingKey): - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Body = Body.decode('utf-8') - Body = json.loads(Body) - body = DictToObject(Body) - if ServerId == body.ServerId: - ret, errmsg = WriteDiskId(body.Path, body.Id) - if ret is False: - ResponseReturn = mqmanage.mq.MqReturn(ret, Code=1, Messages=errmsg) - Response.IsProcessed = True - logger.debug(ResponseReturn) - return ResponseReturn - elif RoutingKey.endswith(RoutKeyDiskAdded): - GlobalFlag['DiskUpdated'] = Updated - logging.log(logging.INFO, "Disk Info is Added") - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Body = Body.decode('utf-8') - Body = json.loads(Body) - body = DictToObject(Body) - Response.IsProcessed = True - logger.debug(ResponseReturn) - return ResponseReturn - elif RoutingKey.endswith(RoutKeyDiskPoolUpdate) or \ - RoutingKey.endswith(RoutKeyDiskStartStop) or \ - RoutingKey.endswith(RoutKeyDiskState) or \ - RoutingKey.endswith(RoutKeyDiskDel): - logging.log(logging.INFO, "Disk Info is Updated") - GlobalFlag['DiskUpdated'] = Updated - GlobalFlag['DiskPoolUpdated'] = Updated - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Body = Body.decode('utf-8') - Body = json.loads(Body) - body = DictToObject(Body) - Response.IsProcessed = True - logger.debug(ResponseReturn) - #UpdateDiskPoolXml() - return ResponseReturn - else: - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Response.IsProcessed = True - return ResponseReturn - @catch_exceptions() def UpdateDiskPoolXml(): diff --git a/core/common/disk/diskpool_manage.py b/core/common/disk/diskpool_manage.py index 65058ef2..0a0cff76 100644 --- a/core/common/disk/diskpool_manage.py +++ b/core/common/disk/diskpool_manage.py @@ -12,18 +12,18 @@ import os, sys -import pdb -import time if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.httpapi import * from const.common import * -from common.utils import Byte2HumanValue, DisplayDiskState, DisplayDiskMode -from const.disk import RequestDiskPool, DiskPoolDetailModule, DiskPoolItemsModule, DiskAdd2DiskPoolItems -from const.http import ResponseHeaderModule -from server.server_manage import GetAllServerDetailInfo -import jsonpickle +from const.disk import DiskAdd2DiskPoolItems, RequestDiskPool +from const.common import DiskPoolDetailModule, DiskPoolItemsModule +from common.utils import * +from common.base_utils import * +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from const.http import ResponseHeaderModule @catch_exceptions() @@ -207,7 +207,7 @@ def GetDiskPoolInfo(Ip, Port, ApiKey, DiskPoolId=None, DiskPoolName=None, logge ReturnType = DiskPoolDetailModule ItemsHeader = False else: - Url = "/api/v1/DiskPools" + Url = "/api/v1/DiskPools/Details" ReturnType = DiskPoolItemsModule ItemsHeader = True @@ -250,194 +250,152 @@ def GetAllDiskPoolListDetail(Ip, Port, logger=None): return Res, Errmsg, None, None -@catch_exceptions() -def ShowDiskPoolInfo(DiskPoolList, ServerDetailInfo, Detail=False, SysinfoDisp=False): - """ - Display Disk list - :param DiskList: DiskItems object list - :param ServerId: - :param DiskId: - :param Detail: - :return: - """ - DiskPools = list() +def ShowDiskPoolInfoNew(DiskPoolList, Detail=SimpleInfo, SysinfoDsp=False): - for pool in DiskPoolList: - TmpPool = dict() - TmpPool['Name'] = pool.Name - TmpPool['Id'] = pool.Id - TmpPool['Description'] = pool.Description - TmpPool['DiskPoolType'] = pool.DiskPoolType - TmpPool['ReplicationType'] = pool.ReplicationType - TmpPool['DiskList'] = list() - DiskPools.append(TmpPool) - - for pool in DiskPools: - PoolId = pool['Id'] - for svr in ServerDetailInfo: - ServerName = svr.Name - for disk in svr.Disks: - if disk.DiskPoolId != PoolId: - continue - - TmpDisk = dict() - TmpDisk['DiskName'] = disk.Name - TmpDisk['DiskId'] = disk.Id - TmpDisk['ServerName'] = ServerName - TmpDisk['Path'] = disk.Path - TmpDisk['TotalSize'] = disk.TotalSize - TmpDisk['UsedSize'] = disk.UsedSize - TmpDisk['Read'] = disk.Read - TmpDisk['Write'] = disk.Write - TmpDisk['State'] = disk.State - TmpDisk['RwMode'] = disk.RwMode - pool['DiskList'].append(TmpDisk) - - if SysinfoDisp is True: + if SysinfoDsp is True: TopTitleLine = "%s" % ("=" * 105) DiskPoolTitleLine = "%s" % ("-" * 105) print(TopTitleLine) DiskPoolTitle = "|%s|%s|%s|%s|" % ( - 'Diskpool'.center(25), 'DiskPoolType'.center(15), 'ReplicationType'.center(17), " " * 43) + 'DiskPoolName'.ljust(26), 'DiskPoolId'.ljust(38), 'DiskPoolType'.ljust(18), 'Tolerance'.ljust(18)) print(DiskPoolTitle) print(TopTitleLine) - DiskTitleLine = "%s%s" % (" " * 4, "-" * 101) - for pool in DiskPools: - _pool = "|%s|%s|%s|%s|" % (pool['Name'].center(25), - str(pool['DiskPoolType']).center(15), - GetReplicationDspType(str(pool['ReplicationType'])).center(17), " " * 43) - print(_pool) - print(DiskPoolTitleLine) - DiskTitle = "%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 4, 'Disk'.center(21), 'Server'.center(15), - 'DiskPath'.center(17), 'TotalSize'.center(11), - 'UsedSize'.center(10), - 'RwMode'.center(12), - 'State'.center(7)) - print(DiskTitle) - - if len(pool['DiskList']) > 0: - print(DiskTitleLine) - for idx, disk in enumerate(pool['DiskList']): - _disk = "%s|%s|%s|%s|%s|%s|%s|%s|" % ( - ' ' * 4, disk['DiskName'].center(21), disk['ServerName'].center(15), - disk['Path'].center(17), str(Byte2HumanValue(int(disk['TotalSize']), 'TotalSize', Color=False)).center(11), - str(Byte2HumanValue(int(disk['UsedSize']), 'UsedSize', Color=False)).center(10), - str(disk['RwMode']).center(12), - str(disk['State']).center(7)) - - print(_disk) - if len(pool['DiskList']) - 1 == idx: - print(DiskPoolTitleLine) - else: - print(DiskTitleLine) - else: - print(DiskTitleLine) - print("%s|%s|" % (' ' * 4, 'No disk data'.center(99))) - print(DiskPoolTitleLine) + DiskTitle = "%s|%s|%s|%s|%s|%s|" % (' ' * 27, 'ServerName'.ljust(19), '' 'DiskName'.ljust(20), + 'DiskPath'.ljust(20), 'RwMode'.ljust(6), 'Status'.ljust(7)) + + DiskTitleLine = "%s%s" % (" " * 27, "-" * 78) + else: - if Detail in [DetailInfo, MoreDetailInfo]: - TopTitleLine = "%s" % ("=" * 173) - DiskPoolTitleLine = "%s" % ("-" * 173) + if Detail == DetailInfo: + TopTitleLine = "%s" % ("=" * 177) + DiskPoolTitleLine = "%s" % ("-" * 177) print(TopTitleLine) - DiskPoolTitle = "|%s|%s|%s|%s|%s|" % ('Name'.center(26), 'PoolId'.center(38), 'DiskPoolType'.center(15), 'ReplicationType'.center(15), " " * 73) + DiskPoolTitle = "|%s|%s|%s|%s|%s|" % ( + 'DiskPoolName'.ljust(26), 'DiskPoolId'.ljust(38), 'DiskPoolType'.ljust(18), 'Tolerance'.ljust(18), " " * 71) print(DiskPoolTitle) print(TopTitleLine) - DiskTitleLine = "%s%s" % (" " * 27, "-" * 146) - for pool in DiskPools: - _pool = "|%s|%s|%s|%s|%s|" % (pool['Name'].center(26), str(pool['Id']).center(38), - str(pool['DiskPoolType']).center(15), GetReplicationDspType(str(pool['ReplicationType'])).center(15), " " * 73) - print(_pool) - print(DiskPoolTitleLine) - DiskTitle = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 27, 'HostName'.center(15), '' 'DiskName'.center(20), - 'DiskPath'.center(20), 'TotalSize'.center(9), 'UsedSize'.center(8), - 'Read'.center(6), 'Write'.center(6), 'RwMode'.center(6), 'State'.center(7), 'DiskId'.center(38)) - print(DiskTitle) - if len(pool['DiskList']) > 0: - print(DiskTitleLine) - for idx, disk in enumerate(pool['DiskList']): - _disk = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 27, disk['ServerName'].center(15), disk['DiskName'].center(20), - disk['Path'].center(20), str(Byte2HumanValue(int(disk['TotalSize']), 'TotalSize')).center(9), - str(Byte2HumanValue(int(disk['UsedSize']), 'UsedSize')).center(8), str(Byte2HumanValue(int(disk['Read']), 'DiskRw')).center(6), - str(Byte2HumanValue(int(disk['Write']), 'DiskRw')).center(6), DisplayDiskMode(disk['RwMode']).center(6), DisplayDiskState(disk['State']).center(7), disk['DiskId'].center(38)) - - print(_disk) - if len(pool['DiskList']) - 1 == idx: - print(DiskPoolTitleLine) - else: - print(DiskTitleLine) - else: - print(DiskTitleLine) - print("%s|%s|" % (' ' * 27, 'No disk data'.center(144))) - print(DiskPoolTitleLine) - elif Detail == SimpleInfo: - TopTitleLine = "%s" % ("=" * 135) - DiskPoolTitleLine = "%s" % ("-" * 135) + DiskTitle = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 27, 'ServerName'.ljust(16), '' 'DiskName'.ljust(20), + 'DiskPath'.ljust(20), 'TotalSize'.ljust(9), + 'UsedSize'.ljust(8), + 'Read'.ljust(8), 'Write'.ljust(8), 'RwMode'.ljust(6), + 'Status'.ljust(7), 'DiskId'.ljust(37)) + + DiskTitleLine = "%s%s" % (" " * 27, "-" * 150) + + else: + TopTitleLine = "%s" % ("=" * 139) + DiskPoolTitleLine = "%s" % ("-" * 139) print(TopTitleLine) - DiskPoolTitle = "|%s|%s|%s|%s|%s|" % ('Name'.center(26), 'PoolId'.center(38), 'DiskPoolType'.center(15), 'ReplicationType'.center(15), " " * 35) + DiskPoolTitle = "|%s|%s|%s|%s|%s|" % ( + 'DiskPoolName'.ljust(26), 'DiskPoolId'.ljust(38), 'DiskPoolType'.ljust(18), 'Tolerance'.ljust(18), " " * 33) print(DiskPoolTitle) print(TopTitleLine) - DiskTitleLine = "%s%s" % (" " * 28, "-" * 107) - for pool in DiskPools: - _pool = "|%s|%s|%s|%s|%s|" % (pool['Name'].center(26), str(pool['Id']).center(38), - str(pool['DiskPoolType']).center(15), GetReplicationDspType(str(pool['ReplicationType'])).center(15), " " * 35) - print(_pool) - print(DiskPoolTitleLine) - DiskTitle = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 27, 'HostName'.center(16), '' 'DiskName'.center(20), - 'DiskPath'.center(20), 'TotalSize'.center(9), 'UsedSize'.center(8), - 'Read'.center(6), 'Write'.center(6), 'RwMode'.center(6), 'State'.center(7)) - print(DiskTitle) - - if len(pool['DiskList']) > 0: - print(DiskTitleLine) - for idx, disk in enumerate(pool['DiskList']): - _disk = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 27, disk['ServerName'].center(16), disk['DiskName'].center(20), - disk['Path'].center(20), str(Byte2HumanValue(int(disk['TotalSize']), 'TotalSize')).center(9), - str(Byte2HumanValue(int(disk['UsedSize']), 'UsedSize')).center(8), str(Byte2HumanValue(int(disk['Read']), 'DiskRw')).center(6), - str(Byte2HumanValue(int(disk['Write']), 'DiskRw')).center(6), DisplayDiskMode(disk['RwMode']).center(6), DisplayDiskState(disk['State']).center(7)) - - print(_disk) - if len(pool['DiskList']) - 1 == idx: - print(DiskPoolTitleLine) - else: - print(DiskTitleLine) + DiskTitle = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 27, 'ServerName'.ljust(16), '' 'DiskName'.ljust(20), + 'DiskPath'.ljust(20), 'TotalSize'.ljust(9), + 'UsedSize'.ljust(8), + 'Read'.ljust(8), 'Write'.ljust(8), 'RwMode'.ljust(6), + 'Status'.ljust(7)) + DiskTitleLine = "%s%s" % (" " * 27, "-" * 112) + + for pool in DiskPoolList: + if SysinfoDsp is True: + _pool = "|%s|%s|%s|%s|" % (pool.Name.ljust(26), pool.Id.ljust(38), + str(pool.DiskPoolType).ljust(18), + GetReplicationDspType(str(pool.ReplicationType), EC=pool.EC).ljust(18)) + else: + if Detail == DetailInfo: + _pool = "|%s|%s|%s|%s|%s|" % (pool.Name.ljust(26), pool.Id.ljust(38), + str(pool.DiskPoolType).ljust(18), GetReplicationDspType(str(pool.ReplicationType), EC=pool.EC).ljust(18),' '* 71) + else: + _pool = "|%s|%s|%s|%s|%s|" % (pool.Name.ljust(26), pool.Id.ljust(38), + str(pool.DiskPoolType).ljust(18), GetReplicationDspType(str(pool.ReplicationType), EC=pool.EC).ljust(18), + " " * 33) + + print(_pool) + print(DiskPoolTitleLine) + print(DiskTitle) + + TotalDiskList = list() + ServerNameDict = dict() + # ordering severname and disk name of diskpool + for server in pool.Servers: + servername = server['Name'] + if servername not in ServerNameDict: + ServerNameDict[servername] = list() + DiskNameDict = dict() + for disk in server['Disks']: + diskname = disk['Name'] + disk['ServerName'] = servername + DiskNameDict[diskname] = disk + + for diskname in sorted(DiskNameDict.keys(), key=str.casefold): # ordering disk name + diskinfo = DiskNameDict[diskname] + ServerNameDict[servername].append(diskinfo) + + for server in sorted(ServerNameDict.keys(), key=str.casefold): # ordering server name + TotalDiskList += ServerNameDict[server] + + if len(TotalDiskList) > 0: + print(DiskTitleLine) + for idx, disk in enumerate(TotalDiskList): + disk = DictToObject(disk) + + if SysinfoDsp is True: + _disk = "%s|%s|%s|%s|%s|%s|" % ( + ' ' * 27, disk.ServerName.ljust(19), disk.Name.ljust(20), + disk.Path.ljust(20), DisplayDiskMode(disk.RwMode).ljust(6), + DisplayDiskState(disk.State).ljust(7)) else: - print(DiskTitleLine) - print("%s|%s|" % (' ' * 28, 'No disk data'.center(105))) + if Detail == DetailInfo: + _disk = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ( + ' ' * 27, disk.ServerName.ljust(16), disk.Name.ljust(20), + disk.Path.ljust(20), str(Byte2HumanValue(int(disk.TotalSize), 'TotalSize')).rjust(9), + str(Byte2HumanValue(int(disk.UsedSize), 'UsedSize')).rjust(8), + str(Byte2HumanValue(int(disk.Read), 'DiskRw')).rjust(8), + str(Byte2HumanValue(int(disk.Write), 'DiskRw')).rjust(8), + DisplayDiskMode(disk.RwMode).ljust(6), DisplayDiskState(disk.State).ljust(7), disk.Id.ljust(37)) + else: + _disk = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ( + ' ' * 27, disk.ServerName.ljust(16), disk.Name.ljust(20), + disk.Path.ljust(20), str(Byte2HumanValue(int(disk.TotalSize), 'TotalSize')).rjust(9), + str(Byte2HumanValue(int(disk.UsedSize), 'UsedSize')).rjust(8), + str(Byte2HumanValue(int(disk.Read), 'DiskRw')).rjust(8), + str(Byte2HumanValue(int(disk.Write), 'DiskRw')).rjust(8), + DisplayDiskMode(disk.RwMode).ljust(6), DisplayDiskState(disk.State).ljust(7)) + + print(_disk) + if len(TotalDiskList) - 1 == idx: print(DiskPoolTitleLine) - + else: + print(DiskTitleLine) else: - PoolTitleLine = '%s' % ('=' * 93) - PoolDataLine = '%s' % ('-' * 93) - DiskPoolTitle = "|%s|%s|%s|%s|" % ('Name'.center(20), 'PoolId'.center(38), 'DiskPoolType'.center(15), 'ReplicationType'.center(15)) - print(PoolTitleLine) - print(DiskPoolTitle) - print(PoolTitleLine) - - if len(DiskPoolList) > 0: - for pool in DiskPoolList: - _pool = "|%s|%s|%s|%s|" % (pool.Name.center(20), str(pool.Id).center(38), - str(pool.DiskPoolType).center(15), GetReplicationDspType(str(pool.ReplicationType)).center(15)) - print(_pool) - print(PoolDataLine) + print(DiskTitleLine) + if SysinfoDsp is True: + print("%s|%s|" % (' ' * 27, 'No disk data'.center(76))) else: - print('No diskpool data') - print(PoolDataLine) + if Detail == DetailInfo: + print("%s|%s|" % (' ' * 27, 'No disk data'.center(148))) + else: + print("%s|%s|" % (' ' * 27, 'No disk data'.center(110))) + print(DiskPoolTitleLine) -def GetReplicationDspType(StringType): +def GetReplicationDspType(StringType, EC=None): if StringType == 'OnePlusOne': - return '1+1' + return 'Replication(1+1)' elif StringType == 'OnePlusZero': - return '1+0' + return 'Disable' elif StringType == 'OnePlusTwo': return '1+2' + elif StringType == 'ErasureCode': + return 'EC(%d:%d)' % (EC['K'], EC['M']) else: return 'Invalid Replica' @@ -458,17 +416,42 @@ def DiskpoolUtilHandler(Conf, Action, Parser, logger): if not (options.DiskpoolName and options.RepType and options.DiskpoolType): Parser.print_help() sys.exit(-1) - if options.RepType not in ['1', '2', 'ec'] or options.DiskpoolType.lower() not in [DiskPoolClassStandard.lower(), DiskPoolClassArchive.lower()]: - Parser.print_help() + + isDiskPoolTypeValid = False + for diskpooltype in ValidDiskPoolType: + if options.DiskpoolType.lower() == diskpooltype.lower(): + isDiskPoolTypeValid = True + break + + if isDiskPoolTypeValid is False: + print('Error : Unsupported DiskPool Type. Valid DiskPool type is %s' % ','.join(ValidDiskPoolType)) sys.exit(-1) - if options.RepType == '1': + + if options.RepType == 'disable': ReplicationType = DiskPoolReplica1 - elif options.RepType == '2': + elif options.RepType == 'replication': ReplicationType = DiskPoolReplica2 else: - #ReplicationType = DiskPoolReplica1 - print('Not supported yet!') - sys.exit(-1) + Ec = ECValueParser.search(options.RepType) + if Ec: + isValid = True + K,M = Ec.groups() + K = int(K) + M = int(M) + if K == 0 or M == 0 or K >= 100 or M >= 100: + print('data chunks(k) or coding chunks(m) must be bigger than zero and less than 100') + isValid = False + elif K < M: + print('coding chunks(m) must be less than or equal to data chunks(k)') + isValid = False + if isValid is False: + sys.exit(-1) + + ReplicationType = 'ErasureCode' + else: + Parser.print_help() + print('Not supported Tolerance type') + sys.exit(-1) Res, Errmsg, Ret = AddDiskPool(PortalIp, PortalPort, PortalApiKey, options.DiskpoolName, options.DiskpoolType, ReplicationType, @@ -560,28 +543,25 @@ def DiskpoolUtilHandler(Conf, Action, Parser, logger): print(Ret.Result, Ret.Message) else: print(Errmsg) + elif Action.lower() == 'list': while True: Res, Errmsg, Ret, DiskPoolList = GetDiskPoolInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) if Res != ResOk: print(Errmsg) else: - if options.MoreDetail: - Detail = MoreDetailInfo - elif options.Detail: + #if options.MoreDetail: + # Detail = MoreDetailInfo + if options.Detail: Detail = DetailInfo else: Detail = SimpleInfo - - Res, Errmsg, Ret, ServerDetailInfo = GetAllServerDetailInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - ShowDiskPoolInfo(DiskPoolList, ServerDetailInfo, Detail=Detail) + ShowDiskPoolInfoNew(DiskPoolList, Detail=Detail) if options.Continue is None: break else: time.sleep(int(options.Continue)) + else: Parser.print_help() diff --git a/core/common/ksanAgent b/core/common/ksanAgent index 092abc75..68e53378 100644 --- a/core/common/ksanAgent +++ b/core/common/ksanAgent @@ -14,26 +14,19 @@ import os, sys import pdb -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.utils import IsDaemonRunning, CreatePidFile -from common.init import UpdateConf, isValidConfig, WaitAgentConfComplete, GetAgentConfig +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) + +#sys.path.insert(1, os.getcwd()) +#sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) from network.network_mq_handle import * -from common.daemon import Daemon -from common.shcommand import UpdateEtcHosts, shcall -from service.service_manage import * -from service.service_mq_handle import ServiceMonitoring +from service.service_manage import ServiceMonitoring from server.server_mq_handle import MonUpdateServerUsage from disk.disk_mq_handle import DiskUsageMonitoring, ReportDiskIo -from const.mq import EdgeRoutingKeyList -from const.common import * from const.network import RequestNetworkInterfaceItems -from mqmanage.mq import * -from rest.server import StartRestServer -import time -from optparse import OptionParser -import threading -import logging -import inspect +from mqmanage.mq import Mq +from rest.agent_api_server import StartRestServer +from const.common import * # Check Network Info @@ -248,7 +241,7 @@ def RegisterManagementNetwork(logger): break if isNetworkAdded is True: - RegisterService(conf, TypeServiceAgent, logger) + RegisterService(conf, KsanAgentName, logger, ServiceNicname='Agent') diff --git a/core/common/ksanEdge b/core/common/ksanEdge deleted file mode 100644 index 60d9c13d..00000000 --- a/core/common/ksanEdge +++ /dev/null @@ -1,287 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import os, sys -import pdb - -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.utils import IsDaemonRunning, CreatePidFile -from common.init import GetConf, InitMonServicedConf, UpdateConf, GetHostInfo -from network.network_mq_handle import * -from common.daemon import Daemon -from common.shcommand import UpdateEtcHosts, shcall -from service.service_manage import * -from mqmanage.mq import * -import time -from optparse import OptionParser -import threading -import logging - -# Check Network Info - - -def AddNewRoutingKeyWithServerId(conf): - """ - Add Rpc routing Key to Routing Key List - """ - EdgeRoutingKeyList.append("*.servers.%s.interfaces.check" % conf.mgs.ServerId) - EdgeRoutingKeyList.append("*.servers.%s.interfaces.update" % conf.mgs.ServerId) - - EdgeRoutingKeyList.append("*.servers.%s.disks.write_disk_id" % conf.mgs.ServerId) - EdgeRoutingKeyList.append("*.servers.%s.disks.check_mount" % conf.mgs.ServerId) - - EdgeRoutingKeyList.append("*.services.%s.config.gw.load" % conf.mgs.ServerId) - EdgeRoutingKeyList.append("*.services.%s.config.gw.save" % conf.mgs.ServerId) - EdgeRoutingKeyList.append("*.services.%s.config.osd.load" % conf.mgs.ServerId) - EdgeRoutingKeyList.append("*.services.%s.config.osd.save" % conf.mgs.ServerId) - EdgeRoutingKeyList.append("*.services.%s.control" % conf.mgs.ServerId) - -def MqHandler(conf, GlobalFlag, logger=None): - ServiceList = dict() - LocalIpList = list() - ServiceList['IdList'] = list() - ServiceList['Details'] = dict() - LoadServiceList(conf, ServiceList, LocalIpList, logger) - - AddNewRoutingKeyWithServerId(conf) - - mq = Mq(conf.mgs.PortalIp, conf.mgs.MqPort, '/', conf.mgs.MqUser, conf.mgs.MqPassword, - EdgeRoutingKeyList, ExchangeName, - 'IfsEdge-%s' % conf.mgs.ServerId, ServerId=conf.mgs.ServerId, ServiceList=ServiceList, logger=logger, MonConf=conf, GlobalFlag=GlobalFlag) - mq.Receiver() - - - -def Start(LocalIp, logger=None, Foreground=False): - """ - LocalIp is used for checking /etc/hosts right after register server. - local server info can be fail to get right after regster. - :param conf: - :param logger: - :param Foreground: - :param LocalIp: - :return: - """ - ret, conf = GetConf(MonServicedConfPath) - if ret is False: - print('Init conf first') - sys.exit(-1) - else: - if conf.mgs.ServerId == '': - print('Local server is not registered. Excute ksanEdge register first') - sys.exit(-1) - - if Foreground is False: - D = Daemon(KsanEdgePidFile, 'ksanEdge') - D.start() - - - GlobalFlag = {'ServiceUpdated': Checked, 'NetworkUpdated': Checked, 'DiskUpdated': Checked, 'DiskPoolUpdated': Checked} - - th = threading.Thread(target=RegisterManagementNetwork, args=(conf,logger, )) - th.start() - - th = threading.Thread(target=KsanServiceRegister, args=(conf, TypeServiceEdge, logger, )) - th.start() - - th = threading.Thread(target=EdgeCheckEtcHosts, args=(conf, logger, LocalIp,)) - th.start() - - Pid = os.getpid() - CreatePidFile(Pid, KsanEdgePidFile) - MqHandler(conf, GlobalFlag, logger=logger) - -def Stop(): - D = Daemon(KsanEdgePidFile, 'ksanEdge') - D.stop() - - -@catch_exceptions() -def RegisterManagementNetwork(conf, logger): - time.sleep(5) - netif = GetNetwork(logger=logger) - nicinfo = netif.GetNicInfoWithNicName(conf.mgs.ManagementNetDev) - netif = RequestNetworkInterfaceItems(conf.mgs.ManagementNetDev, Description='Management Network', logger=logger) - netif.Set(nicinfo) - isNetworkAdded = False - RetryCnt = 0 - while True: - RetryCnt += 1 - Res, Errmsg, Ret = AddNetworkInterface(conf.mgs.PortalIp, int(conf.PortalPort), conf.PortalApiKey, conf.ServerId, - netif, logger=logger) - if Res != ResOk: - logger.error('fail to add Network Interface %s retry:%d' %(Errmsg, RetryCnt)) - else: - NetworkInterfaceId = None - if Ret.Result == ResultSuccess : - logging.log(logging.INFO, "success to add network %s %s" % (str(Ret.Result), str(Ret.Message))) - NetworkInterfaceId = Ret.Data.Id - elif Ret.Code == CodeDuplicated: - Res, Errmsg, Ret, Data = GetNetworkInterface(conf.mgs.PortalIp, int(conf.mgs.PortalPort), conf.mgs.PortalApiKey, - conf.mgs.ServerId, InterfaceId=None, disp=False, logger=logger) - if Res != ResOk or Ret.Result != ResultSuccess: - logging.log(logging.INFO, "%s %s" % (str(Ret.Result), str(Ret.Message))) - else: - NetworkInterfaceId = Data[0].Id - else: - logger.error('fail to add Network Interface' + Ret.Message) - if NetworkInterfaceId is not None: - UpdateConf(MonServicedConfPath, 'mgs', 'DefaultNetworkId', NetworkInterfaceId) - isNetworkAdded = True - break - if RetryCnt <= 5: - time.sleep(5) - else: - logger.error('fail to add Network Interface. ' + Ret.Message) - break - - if isNetworkAdded is True: - RegisterService(conf, TypeServiceEdge, logger) - - - -def GetValidEtcHostsInfo(Data): - ValidHostsInfo = list() - for svr in Data: - try: - if len(svr.NetworkInterfaces) > 0: - ManagementIp = svr.NetworkInterfaces[0].IpAddress - else: - continue - HostName = svr.Name - ValidHostsInfo.append((ManagementIp, HostName)) - except Exception as err: - print(err) - return ValidHostsInfo - - -def EdgeCheckEtcHosts(conf, logger, LocalIp): - """ - check current serverlist and check /etc/hosts with the server info and update if not applied. - :param conf: - :return: - """ - LocalHostInfo = list() - if LocalIp is not None: # run by ksanServerRegister - out, err = shcall('hostname') - LocalHostName = out[:-1] - LocalHostInfo.append((LocalIp, LocalHostName)) - - Res, Errmsg, Ret, Data = GetAllServerDetailInfo(conf.mgs.PortalIp, int(conf.mgs.PortalPort), conf.mgs.PortalApiKey, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - ValidHostsInfo = GetValidEtcHostsInfo(Data) - ValidHostsInfo += LocalHostInfo - UpdateEtcHosts(ValidHostsInfo, 'add') - else: - logger.error('fail to get server info %s' % Errmsg) - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanEdge {start|stop|status} [option] - start : Start ksanEdge - stop : Stop ksanEdge - status : Status ksanEdge - [options] - -h, --help : show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {start|stop|status} [option]" - parser = MyOptionParser(usage=usage) - #parser.add_option('-S', "--Server", action='store_true', - # dest='Server', default=False, help='Server Option') - parser.add_option('-S', "--ServerId", dest="id", help='server id') - parser.add_option('-i', "--LocalIp", dest="ip", default=None, help='Local Ip Address') - parser.add_option('-D', "--Description", dest="desc", default='', help='server description') - parser.add_option('-N', "--Name", dest="NicName", default=None, help='Network Interface Name') - parser.add_option('-T', "--Detail", dest="Detail", default=None, help='server info in detail') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='Debug mode') - parser.add_option('-f', "--Foreground", dest="Foreground", action='store_true', default=False, help='Running as Foreground') - - options, args = parser.parse_args() - - if len(args) != 1: - parser.print_help() - sys.exit(-1) - - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - else: - logger = Logging(loglevel='error') - logger = logger.create() - - ret, conf = GetConf(MonServicedConfPath) - if ret is False and args[0] != 'init': - print('Check if configuration is done') - sys.exit(-1) - - if args[0] == 'init': - InitMonServicedConf() - elif args[0] == 'register': - res, hostname, ip = GetHostInfo() - if res is not True: - print('fail to get HostName and Ip, %s' % hostname) - sys.exit(-1) - Res, Errmsg, Ret, ServerInfo = RegisterServer(conf.mgs.PortalIp, int(conf.mgs.PortalPort), conf.mgs.PortalApiKey, options.desc, - Name=hostname, logger=logger) - if Res != ResOk: - print('fail to add Server' + Errmsg) - else: - if Ret.Result == ResultSuccess or Ret.Code == CodeDuplicated: - UpdateConf(MonServicedConfPath, 'mgs', 'ServerId', ServerInfo.Id) - print('Done') - else: - print('fail to add Server' + Ret.Message) - - elif args[0] == 'unregister': - if options.id is None: - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = RemoveServer(conf.mgs.PortalIp, int(conf.mgs.PortalPort), conf.mgs.PortalApiKey, ServerId=options.id, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - elif args[0] == 'start': - Ret, Pid = IsDaemonRunning(KsanEdgePidFile, 'ksanEdge') - if Ret is True: - print('Already running') - else: - Start(options.ip, logger=logger, Foreground=options.Foreground) - print('Done') - - elif args[0] == 'stop': - Ret, Pid = IsDaemonRunning(KsanEdgePidFile, 'ksanEdge') - if Ret is False: - print('ksanEdge is not running') - else: - Stop() - print('Done') - - - elif args[0] == 'status': - print('KsanEdge ... ',end=' ') - Ret, Pid = IsDaemonRunning(KsanEdgePidFile, 'ksanEdge') - if Ret is True: - print('Ok') - else: - print('Not Ok') - else: - parser.print_help() diff --git a/core/common/ksanMon b/core/common/ksanMon deleted file mode 100644 index a7ac6f7f..00000000 --- a/core/common/ksanMon +++ /dev/null @@ -1,154 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - - -import os, sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.daemon import Daemon -from network.network_mq_handle import * -from disk.disk_mq_handle import DiskUsageMonitoring -from server.server_mq_handle import * -from service.service_mq_handle import * -from mqmanage.mq import Mq -from common.init import GetConf -from common.utils import CreatePidFile -from common.define import MonRoutingKeyList - -from optparse import OptionParser -import threading - -# Check Network Info - - -# Check Disk Info - -def MqHandler(conf, GlobalFlag, logger): - ServiceList = dict() - LocalIpList = list() - ServiceList['IdList'] = list() - ServiceList['Details'] = dict() - LoadServiceList(conf, ServiceList, LocalIpList, logger) - mq = Mq(conf.mgs.PortalIp, conf.mgs.MqPort, '/', conf.mgs.MqUser, conf.mgs.MqPassword, MonRoutingKeyList, - ExchangeName, 'IfsMon-%s' % conf.mgs.ServerId, ServiceList=ServiceList, - ServerId=conf.mgs.ServerId, GlobalFlag=GlobalFlag, logger=logger) - mq.Receiver() - - -def Start(Foreground=False): - ret, conf = GetConf(MonServicedConfPath) - if ret is False: - print('Init conf first') - sys.exit(-1) - else: - if conf.mgs.ServerId == '': - print('Local server is not registered. Excute ksanEdge register first') - sys.exit(-1) - - if Foreground is False: - D = Daemon(KsanMonPidFile, 'ksanMon') - D.start() - - logger = Logging().create() - - GlobalFlag = {'ServiceUpdated': Checked, 'NetworkUpdated': Checked, 'DiskUpdated': Checked, 'DiskPoolUpdated': Checked} - - threads = list() - th = threading.Thread(target=DiskUsageMonitoring, args=(conf, GlobalFlag, logger, )) - th.start() - threads.append(th) - - th = threading.Thread(target=KsanServiceRegister, args=(conf, TypeServiceMonitor, logger, )) - th.start() - threads.append(th) - - th = threading.Thread(target=UpdateNetworkStat, args=(conf, GlobalFlag, logger, )) - th.start() - threads.append(th) - - th = threading.Thread(target=MonUpdateServerUsage, args=(conf, logger, )) - th.start() - threads.append(th) - - th = threading.Thread(target=ServiceMonitoring, args=(conf, GlobalFlag, logger, )) - th.start() - threads.append(th) - - Pid = os.getpid() - CreatePidFile(Pid, KsanMonPidFile) - MqHandler(conf, GlobalFlag, logger) - for th in threads: - th.join() - -def Stop(): - D = Daemon(KsanMonPidFile, 'ksanMon') - D.stop() - - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanMon {start|stop|satatus} - start : Start ksanMon - stop : Stop ksanMon - [options] - -h, --help : show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {start|stop|status}" - parser = MyOptionParser(usage=usage) - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='Debug mode') - parser.add_option('-f', "--Foreground", dest="Foreground", action='store_true', default=False, help='Running as Foreground') - - options, args = parser.parse_args() - - if len(args) != 1: - parser.print_help() - sys.exit(-1) - - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - - ret, conf = GetConf(MonServicedConfPath) - #if ret is False and args[0] != 'init': - # print('Check if configuration is done') - # sys.exit(-1) - - if args[0] == 'start': - Ret, Pid = IsDaemonRunning(KsanMonPidFile, 'ksanMon') - if Ret is True: - print('ksanMon is already running') - else: - Start(Foreground=options.Foreground) - - elif args[0] == 'stop': - Ret, Pid = IsDaemonRunning(KsanMonPidFile, 'ksanMon') - if Ret is False: - print('ksanMon is not running') - else: - Stop() - - elif args[0] == 'status': - print('KsanMon ... ',end=' ') - Ret, Pid = IsDaemonRunning(KsanMonPidFile, 'ksanMon') - if Ret is True: - print('Ok') - else: - print('Not Ok') - else: - parser.print_help() diff --git a/core/common/ksanMongoDBManager b/core/common/ksanMongoDBManager deleted file mode 100644 index 09a641f2..00000000 --- a/core/common/ksanMongoDBManager +++ /dev/null @@ -1,523 +0,0 @@ -#!/usr/bin/python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import os, sys - -from subprocess import Popen, PIPE -import socket -from optparse import OptionParser -from common.shcommand import UpdateEtcHosts -from common.utils import CreatePidFile -#from common.log import Logging -import logging -import json -import re -import time - -MongDbCluster = {'Shard1Port': 20001, 'Shard2Port': 20002, 'ConfigServerPort': 50000, 'MongoDbPort': 27017} - -def shcall(command): - """ - Executes a command and returns the result - """ - try: - p = Popen(command, shell=True, stdout=PIPE, stderr=PIPE, close_fds=True, universal_newlines=True) - except (OSError, ValueError) as err: - return '', err - return p.communicate() - - -def GetHostInfo(): - try: - hostname = socket.gethostname() - ip = socket.gethostbyname(hostname) - return True, hostname, ip - except socket.error as err: - return False, str(err), None - - -MongoVersion = '4.4' -MongoYumRepoFile = '/etc/yum.repos.d/mongodb-org-%s.repo' % MongoVersion -MongoDbLogPath = '/var/log/ksan/mongodb' -MongoDbHomeDir = '/var/lib/mongo' -KsanMongoPath = '/usr/local/ksan/mongo' - -ShardStartCmd = "%s/shard.start" % KsanMongoPath -ShardStopCmd = "%s/shard.stop" % KsanMongoPath -ConfigdStartCmd = "%s/configd.start" % KsanMongoPath -ConfigdStopCmd = "%s/configd.stop" % KsanMongoPath -MongosStartCmd = "%s/mongos.start" % KsanMongoPath -MongosStopCmd = "%s/mongos.stop" % KsanMongoPath - - -def InstallMongo(): - with open(MongoYumRepoFile, 'w') as f: - YumConf = """ -[mongodb-org-%s] -name=MongoDB Repository -baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/%s/x86_64/ -gpgcheck=1 -enabled=1 -gpgkey=https://www.mongodb.org/static/pgp/server-%s.asc -sslverify=0 - """ % (MongoVersion, MongoVersion, MongoVersion) - f.write(YumConf) - YumCmd = "yum clean all; yum repolist; yum install -y mongodb-org" - out, err = shcall(YumCmd) - -def SetMongosCmd(Shard1Port, Shard2Port, ConfigServerPort, MongoDbPort): - if not os.path.exists(KsanMongoPath): - os.makedirs(KsanMongoPath) - logging.log(logging.INFO, "Create Mongo Script with Shard1port:%s Shard2Port:%s ConfigServerPort:%s" % - (Shard1Port, Shard2Port, ConfigServerPort)) - # create start shard command - with open(ShardStartCmd, 'w') as f: - Contents = """ - #Shard1Port:%s - #Shard2Port:%s - mongod --shardsvr --bind_ip_all --dbpath %s/shard1 --port %s --replSet reps%s --logpath %s/shard1 & - mongod --shardsvr --bind_ip_all --dbpath %s/shard2 --port %s --replSet reps%s --logpath %s/shard2 & - """ % (Shard1Port, Shard2Port, MongoDbHomeDir, Shard1Port, Shard1Port, MongoDbLogPath, MongoDbHomeDir, - Shard2Port, Shard2Port, MongoDbLogPath) - f.write(Contents) - - # create stop shard command - with open(ShardStopCmd, 'w') as f: - Contents = "kill -9 $(ps -ef |grep \"mongod --shardsvr \" |grep -v grep|awk '{print $2}')" - f.write(Contents) - - # create start configd command - with open(ConfigdStartCmd, 'w') as f: - Contents = """ - #ConfigServerPort:%s - mongod --configsvr --bind_ip_all --dbpath %s/configd --port %s --replSet repsc --logpath %s/configdb & - """ % (ConfigServerPort, MongoDbHomeDir, ConfigServerPort, MongoDbLogPath) - f.write(Contents) - - # create stop configd command - with open(ConfigdStopCmd, 'w') as f: - Contents = "kill -9 $(ps -ef |grep \"mongod --configsvr \" |grep -v grep|awk '{print $2}')" - f.write(Contents) - # create start mongos command - with open(MongosStartCmd, 'w') as f: - Contents = """ - #MongoDbPort:%s - mongos --configdb repsc/localhost:%s --port %s --bind_ip_all --logpath %s/mongos --pidfilepath /var/run/mongod.pid & - """ % (MongoDbPort, ConfigServerPort, MongoDbPort, MongoDbLogPath) - f.write(Contents) - - # create stop mongos command - with open(MongosStopCmd, 'w') as f: - Contents = "kill -9 $(ps -ef |grep \"mongos --configdb \" |grep -v grep|awk '{print $2}')" - f.write(Contents) - - # change executable scripts mode - out ,err = shcall("chmod +x /usr/local/ksan/mongo/*") - print(out, err) - -def StartMongoDb(): - os.system(ShardStartCmd) - os.system(ConfigdStartCmd) - os.system(MongosStartCmd) - -def StopMongoDb(): - out, err = shcall(ShardStopCmd) - print(out, err) - out, err = shcall(ConfigdStopCmd) - print(out, err) - out, err = shcall(MongosStopCmd) - print(out, err) - -def StatusMongoDb(): - pass - - -def GetClusterMemberList(HostsArgs): - ArgHostsList = HostsArgs.split() - MemberList = list() - for HostName in ArgHostsList: - try: - Ip = socket.gethostbyname(HostName) - except socket.gaierror as err: - print('Invalid hostname. Check /etc/hosts') - return None - else: - MemberList.append((Ip, HostName)) - - if len(ArgHostsList) != len(MemberList): - print("Invalid member type is found in member argument") - return MemberList - - -def InitMongos(Option): - global MongoDbHomeDir - """ - if Option.ClusterMembers: - ClusterMembers = GetClusterMemberList(Option.ClusterMembers) - else: - ClusterMembers = list() - if ClusterMembers is None: - return - """ - Shard1Port = Option.Shard1Port - Shard2Port = Option.Shard2Port - ConfigServerPort = Option.ConfigServerPort - MongoDbPort = Option.MongoDbPort - if Option.HomeDirectory: - MongoDbHomeDir = Option.HomeDirectory - - # create storage directory - if not os.path.exists(MongoDbHomeDir): - os.makedirs(MongoDbHomeDir) - if not os.path.exists(MongoDbHomeDir + '/configd'): - os.makedirs('%s/configd' % MongoDbHomeDir) - if not os.path.exists(MongoDbHomeDir + '/shard1'): - os.makedirs('%s/shard1' % MongoDbHomeDir) - if not os.path.exists(MongoDbHomeDir + '/shard2'): - os.makedirs('%s/shard2' % MongoDbHomeDir) - if not os.path.exists(MongoDbHomeDir + '/mongo'): - os.makedirs('%s/mongo' % MongoDbHomeDir) - - if not os.path.exists(MongoDbLogPath): - os.makedirs(MongoDbLogPath) - - # set /etc/hosts - #UpdateEtcHosts(ClusterMembers) - - SetMongosCmd(Shard1Port, Shard2Port, ConfigServerPort, MongoDbPort) - if Option.PrimaryNode: - StartMongoDb() - - PrimaryNodeConfigure(Shard1Port, Shard2Port, ConfigServerPort) - RunPrimaryCluster(Shard1Port, Shard2Port) - StopMongoDb() - -def PrimaryNodeConfigure(Shard1Port, Shard2Port, ConfigServerPort): - Retry = 0 - while True: - Retry += 1 - SetClusterCmd = "mongo --port %s --eval 'rs.initiate()';" % Shard1Port - os.system(SetClusterCmd) - CheckCmd = "mongo --port %s --eval 'rs.status()'|grep stateStr" % Shard1Port - out, err = shcall(CheckCmd) - - if "PRIMARY" in out: - break - else: - if Retry > 3: - logging.log(logging.INFO, 'fail to set primary for Shard1') - break - else: - time.sleep(1) - - Retry = 0 - while True: - Retry += 1 - SetClusterCmd = "mongo --port %s --eval 'rs.initiate()';" % Shard2Port - os.system(SetClusterCmd) - CheckCmd = "mongo --port %s --eval 'rs.status()'|grep stateStr" % Shard2Port - out, err = shcall(CheckCmd) - - if "PRIMARY" in out: - break - else: - if Retry > 3: - logging.log(logging.INFO, 'fail to set primary for Shard2') - break - else: - time.sleep(1) - - Retry = 0 - while True: - Retry += 1 - SetClusterCmd = "mongo --port %s --eval 'rs.initiate()';" % ConfigServerPort - os.system(SetClusterCmd) - CheckCmd = "mongo --port %s --eval 'rs.status()'|grep stateStr" % ConfigServerPort - out, err = shcall(CheckCmd) - - if "PRIMARY" in out: - break - else: - if Retry > 3: - logging.log(logging.INFO, 'fail to set primary for ConfigServerPort') - break - else: - time.sleep(1) - - - -def RunPrimaryCluster(Shard1Port, Shard2Port): - out, err = shcall('hostname') - PrimaryHostname = out[:-1] - Retry = 0 - while True: - Retry += 1 - RunCmd = "mongo admin --eval 'db.runCommand({addshard:\"reps%s/%s:%s\"})';" % (Shard1Port, PrimaryHostname, Shard1Port) - os.system(RunCmd) - - CheckCmd = "mongo admin --eval 'sh.status()'|grep reps" - out, err = shcall(CheckCmd) - if 'reps%s' % Shard1Port in out: - break - else: - if Retry > 3: - logging.log(logging.INFO, 'fail to add Shard1') - break - else: - time.sleep(1) - - - Retry = 0 - while True: - Retry += 1 - RunCmd = "mongo admin --eval 'db.runCommand({addshard:\"reps%s/%s:%s\"})';" % (Shard2Port, PrimaryHostname, Shard2Port) - os.system(RunCmd) - - CheckCmd = "mongo admin --eval 'sh.status()'|grep reps" - out, err = shcall(CheckCmd) - if 'reps%s' % Shard2Port in out: - break - else: - if Retry > 3: - logging.log(logging.INFO, 'fail to add Shard2') - break - else: - time.sleep(1) - - -def GetPortInfo(): - global MongDbCluster - isValid = True - if not os.path.exists(ShardStartCmd): - print('Shard Start Command is not found /usr/local/ksan/mongo/') - isValid = False - if not os.path.exists(MongosStartCmd): - print('MongoDB Start Command is not found /usr/local/ksan/mongo/') - isValid = False - - if not os.path.exists(ConfigdStartCmd): - print('ConfigServer Start Command is not found /usr/local/ksan/mongo/') - isValid = False - if isValid is False: - return False - with open(ShardStartCmd, 'r') as f: - Cmd = f.read() - Shard1PortFinder = re.compile('#Shard1Port:([\d]+)') - Shard1Port = Shard1PortFinder.search(Cmd) - if Shard1Port: - MongDbCluster['Shard1Port'] = Shard1Port.groups()[0] - else: - print('Fail to get Shard1 Port') - isValid = False - - Shard2PortFinder = re.compile('#Shard2Port:([\d]+)') - Shard2Port = Shard2PortFinder.search(Cmd) - if Shard2Port: - MongDbCluster['Shard2Port'] = Shard2Port.groups()[0] - else: - print('Fail to get Shard2 Port') - isValid = False - - with open(ConfigdStartCmd, 'r') as f: - Cmd = f.read() - ConfigServerPortFinder = re.compile('#ConfigServerPort:([\d]+)') - ConfigServerPort = ConfigServerPortFinder.search(Cmd) - if ConfigServerPort: - MongDbCluster['ConfigServerPort'] = ConfigServerPort.groups()[0] - else: - print('Fail to get ConfigServer Port') - isValid = False - - with open(MongosStartCmd, 'r') as f: - Cmd = f.read() - MongoDbPortFinder = re.compile('#MongoDbPort:([\d]+)') - MongoDbPort = MongoDbPortFinder.search(Cmd) - if MongoDbPort: - MongDbCluster['MongoDbPort'] = MongoDbPort.groups()[0] - else: - print('Fail to get MongoDB Port') - isValid = False - return isValid - -def AddClusterMember(HostsArgs): - MemberHostNameList = GetClusterMemberList(HostsArgs) - if MemberHostNameList is None: - return - - ret = GetPortInfo() - if ret is False: - return - MongoMemberAddCmd = '' - Shard1Port = MongDbCluster['Shard1Port'] - Shard2Port = MongDbCluster['Shard2Port'] - ConfigServerPort = MongDbCluster['ConfigServerPort'] - - for ip, host in MemberHostNameList: - MongoMemberAddCmd += "mongo --port %s --eval 'rs.add(\"%s:%s\")';sleep 1;" % (Shard1Port, host, Shard1Port) - - for ip, host in MemberHostNameList: - MongoMemberAddCmd += "mongo --port %s --eval 'rs.add(\"%s:%s\")';sleep 1;" % (Shard2Port, host, Shard2Port) - - for ip, host in MemberHostNameList: - MongoMemberAddCmd += "mongo --port %s --eval 'rs.add(\"%s:%s\")';sleep 1;" % (ConfigServerPort, host, ConfigServerPort) - - out, err = shcall(MongoMemberAddCmd) - print(out, err) - - -def RemoveClusterMember(HostsArgs): - MemberHostNameList = GetClusterMemberList(HostsArgs) - if MemberHostNameList is None: - return - - ret = GetPortInfo() - if ret is False: - return - MongoMemberRemoveCmd = '' - Shard1Port = MongDbCluster['Shard1Port'] - Shard2Port = MongDbCluster['Shard2Port'] - ConfigServerPort = MongDbCluster['ConfigServerPort'] - - for ip, host in MemberHostNameList: - MongoMemberRemoveCmd += "mongo --port %s --eval 'rs.remove(\"%s:%s\")';sleep 1;" % (ConfigServerPort, host, ConfigServerPort) - MongoMemberRemoveCmd += "mongo --port %s --eval 'rs.remove(\"%s:%s\")';sleep 1;" % (Shard2Port, host, Shard2Port) - MongoMemberRemoveCmd += "mongo --port %s --eval 'rs.remove(\"%s:%s\")';sleep 1;" % (Shard1Port, host, Shard1Port) - - out, err = shcall(MongoMemberRemoveCmd) - print(out, err) - - -def GetStatus(): - Cmd = "mongo --eval 'db.serverStatus()'" - out, err = shcall(Cmd) - lines = out.split("\n") - JsonOutput = '' - for line in lines: - if 'MongoDB shell' in line or 'connecting to: mongodb' in line or 'Implicit session: session' in line or 'MongoDB server version' in line: - continue - else: - trimlined = line.rsplit() - JsonOutput += ''.join(trimlined) - - HealthFinder = re.compile("state\":\"([\d\w_\-]+)\"") - status = HealthFinder.search(JsonOutput) - if status: - print("%s" % status.groups()[0]) - else: - print(JsonOutput) - -def GetShardStatus(): - out, err = shcall("mongo admin --eval 'sh.status()'|grep -A 2 shards:") - #out = """ - # shards: - # { "_id" : "reps1", "host" : "reps1/ksan_osd1:10000,ksan_osd2:10000", "state" : 1 } - # { "_id" : "reps2", "host" : "reps2/ksan_osd1:20000,ksan_osd2:20000", "state" : 1 } - #""" - if out: - if 'shards:' in out: - lines = out.split('\n') - OutPutString = '' - ShardList = list() - for line in lines: - if '"_id"' in line: - Shard = {'id': '', 'state': '0', 'host': list()} - _Shard = json.loads(line) - Shard['id'] = _Shard['_id'] - HostListString = re.sub('%s/' % Shard['id'], '', _Shard['host']) - Shard['host'] = HostListString.split(',') - Shard['state'] = _Shard['state'] - ShardList.append(Shard) - TitleLine = "-" * 42 - HostLine = "%s%s\n" % (' ' * 20, '-' * 22) - print(TitleLine) - print("|%s|%s|%s|" % ("ID".center(10), "Status".center(8) ,"Host:Port".center(20))) - print(TitleLine) - for shard in ShardList: - IdStat = "|%s|%s" % (shard['id'].center(10), str(shard['state']).center(8)) - Host = '' - for idx, host in enumerate(shard['host']): - if len(Host) == 0: - Host += "|%s|\n" % (host.center(20)) - Host += "%s\n" % TitleLine - else: - Host += "%s|%s|\n" % (" " * 20, host.center(20)) - Host += HostLine - print(IdStat+Host) - print(TitleLine) - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanMongoDBManager {init|start|stop|add_member|remove_member|status} [option] - init -P -S -C - -D -H -R : MongoDb Initialize - - start : Start MongoDb - stop : Stop MongoDb - add_member -M : Add Cluster Members - remove_member -M : Remove Cluster Members - status : Show MongoDb status - status_shard : Show MongoDb Shard status - [options] - -M : Cluster Member. ex) "osd1 osd2" - -P : Shard1 Port - -S : Shard2 Port - -C : Config Server Port - -D : MongoDb Server Port - -H : MongoDb Home Directory - -R : Cluster Primary Node Flag - -h, --help : show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {init|start|start_primary|stop|add_member|remove_member|status} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-M', "--Member", dest="ClusterMembers", help="Cluster Member list. : ex -M 'osd1 osd2'") - parser.add_option('-P', "--Shard1Port", dest="Shard1Port", default='20001', help="Shard1 Port. default 20001") - parser.add_option('-S', "--Shard2Port", dest="Shard2Port", default='20002', help="Shard2 Port. default 20002") - parser.add_option('-C', "--ConfigServerPort", dest="ConfigServerPort", default='50000', help="ConfigServer Port. default 50000") - parser.add_option('-D', "--DbPort", dest="MongoDbPort", default='27017', help="MongoDb Port. default 27017") - parser.add_option('-H', "--HomeDirectory", dest="HomeDirectory", default='/var/lib/mongo', help='MongoDB Configuration Home Directory Path') - parser.add_option('-R', "--PrimaryNode", dest="PrimaryNode", action="store_true", default=False, help='MongoDB Primary Cluster Node Flag') - - options, args = parser.parse_args() - if len(args) != 1: - parser.print_help() - sys.exit(-1) - - if args[0] == 'install': - InstallMongo() - elif args[0] == 'init': - InitMongos(options) - elif args[0] == 'start': - StartMongoDb() - elif args[0] == 'stop': - StopMongoDb() - elif args[0] == 'status': - GetStatus() - elif args[0] == 'status_shard': - GetShardStatus() - elif args[0] == 'add_member': - if not options.ClusterMembers: - parser.print_help() - sys.exit(-1) - AddClusterMember(options.ClusterMembers) - elif args[0] == 'remove_member': - if not options.ClusterMembers: - parser.print_help() - sys.exit(-1) - RemoveClusterMember(options.ClusterMembers) - else: - parser.print_help() diff --git a/core/common/ksanMonitor b/core/common/ksanMonitor deleted file mode 100644 index 1d06922f..00000000 --- a/core/common/ksanMonitor +++ /dev/null @@ -1,162 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import os, sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.daemon import Daemon -from network.network_mq_handle import * -from disk.disk_mq_handle import DiskUsageMonitoring -from server.server_mq_handle import * -from service.service_mq_handle import * -from mqmanage.mq import Mq -from common.init import GetConf -from common.utils import CreatePidFile -from const.mq import MonRoutingKeyList - -from optparse import OptionParser -import threading - -# Check Network Info - - -# Check Disk Info - -def MqHandler(conf, GlobalFlag, logger): - ServiceList = dict() - LocalIpList = list() - ServiceList['IdList'] = list() - ServiceList['Details'] = dict() - LoadServiceList(conf, ServiceList, LocalIpList, logger) - mq = Mq(conf.mgs.PortalIp, conf.mgs.MqPort, '/', conf.mgs.MqUser, conf.mgs.MqPassword, MonRoutingKeyList, - ExchangeName, 'IfsMon-%s' % conf.mgs.ServerId, ServiceList=ServiceList, - ServerId=conf.mgs.ServerId, GlobalFlag=GlobalFlag, logger=logger) - mq.Receiver() - - -def Start(logger, Foreground=False): - ret, conf = GetConf(MonServicedConfPath) - if ret is False: - print('Init conf first') - sys.exit(-1) - else: - if conf.mgs.ServerId == '': - print('Local server is not registered. Excute ksanAgent register first') - sys.exit(-1) - - if Foreground is False: - D = Daemon(KsanMonitorPidFile, 'ksanMonitor') - D.start() - - #logger = Logging().create() - - GlobalFlag = {'ServiceUpdated': Checked, 'NetworkUpdated': Checked, 'DiskUpdated': Checked, 'DiskPoolUpdated': Checked} - - threads = list() - th = threading.Thread(target=DiskUsageMonitoring, args=(conf, GlobalFlag, logger, )) - th.start() - threads.append(th) - - th = threading.Thread(target=KsanServiceRegister, args=(conf, TypeServiceMonitor, logger, )) - th.start() - threads.append(th) - - th = threading.Thread(target=UpdateNetworkStat, args=(conf, GlobalFlag, logger, )) - th.start() - threads.append(th) - - th = threading.Thread(target=MonUpdateServerUsage, args=(conf, logger, )) - th.start() - threads.append(th) - - th = threading.Thread(target=ServiceMonitoring, args=(conf, GlobalFlag, logger, )) - th.start() - threads.append(th) - - Pid = os.getpid() - CreatePidFile(Pid, KsanMonitorPidFile) - MqHandler(conf, GlobalFlag, logger) - for th in threads: - th.join() - -def Stop(): - D = Daemon(KsanMonitorPidFile, 'ksanMonitor') - D.stop() - - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanMonitor {start|stop|satatus} - start : Start ksanMonitor - stop : Stop ksanMonitor - [options] - -h, --help : show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {start|stop|status}" - parser = MyOptionParser(usage=usage) - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='Debug mode') - parser.add_option('-f', "--Foreground", dest="Foreground", action='store_true', default=False, help='Running as Foreground') - - options, args = parser.parse_args() - - if len(args) != 1: - parser.print_help() - sys.exit(-1) - - #logger = None - #if options.debug is True: - # logger = Logging(loglevel='debug') - # logger = logger.create() - - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - else: - logger = Logging(loglevel='error') - logger = logger.create() - - - - ret, conf = GetConf(MonServicedConfPath) - #if ret is False and args[0] != 'init': - # print('Check if configuration is done') - # sys.exit(-1) - - if args[0] == 'start': - Ret, Pid = IsDaemonRunning(KsanMonitorPidFile, 'ksanMonitor') - if Ret is True: - print('ksanMonitor is already running') - else: - Start(logger, Foreground=options.Foreground) - - elif args[0] == 'stop': - Ret, Pid = IsDaemonRunning(KsanMonitorPidFile, 'ksanMonitor') - if Ret is False: - print('ksanMonitor is not running') - else: - Stop() - - elif args[0] == 'status': - print('KsanMon ... ',end=' ') - Ret, Pid = IsDaemonRunning(KsanMonitorPidFile, 'ksanMonitor') - if Ret is True: - print('Ok') - else: - print('Not Ok') - else: - parser.print_help() diff --git a/core/common/ksanNodeRegister b/core/common/ksanNodeRegister deleted file mode 100644 index 7d108f09..00000000 --- a/core/common/ksanNodeRegister +++ /dev/null @@ -1,246 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import os, sys -import pdb - -import common.network -from common.init import read_conf - - -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from service.service_manage import * -from mqmanage.mq import * -from optparse import OptionParser - - -def DependencyCheck(): - isValid = True - try: - import dns.resolver - except Exception as err: - print('dependency dns.resolver is not installed') - isValid = False - - try: - import jsonpickle - except Exception as err: - print('dependency jsonpickle is not installed') - isValid = False - - - try: - import netifaces - except Exception as err: - print('dependency netifaces is not installed') - isValid = False - - - try: - import pika - except Exception as err: - print('dependency pika is not installed') - isValid = False - - - try: - import psutil - except Exception as err: - print('dependency psutil is not installed') - isValid = False - - - try: - import requests - except Exception as err: - print('dependency requests is not installed') - isValid = False - - return isValid - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanAgent [option] - [options] - -i : Local Management Ip - -m : Portal Host - -p : Portal Port - -r : Mq Host - -u : Mq User - -q : Mq Port - -w : Mq Password - -k : Portal Api Key - -h, --help : show this help message and exit -""" - print(Usage) - -def UpdateKsanAgentConfig(PortalIp, PortalPort, LocalIp, MqHost, MqUser, MqPort, MqPassword, ApiKey): - oldconf = None - if os.path.exists(MonServicedConfPath): - ret, oldconf = read_conf(MonServicedConfPath) - if ret is True: - if 'ServerId' in oldconf['mgs']: - if oldconf['mgs']['ServerId'] != '': - return False, 'already registered' - - net = common.network.GetNetwork() - nic = net.GetNicInfo(Ip=LocalIp) - if not nic: - return False, 'fail to get network device with local ip %s' % LocalIp - - NetDevice = nic['Name'] - Conf = """ -[mgs] -PortalHost = %s -PortalPort = %s -MqHost = %s -MqUser = %s -MqPassword = %s -MqPort = %s -PortalApiKey = %s -ServerId = -ManagementNetDev = %s -DefaultNetworkId = -""" % (PortalIp, PortalPort, MqHost, MqUser, MqPassword, MqPort, ApiKey, NetDevice) - if oldconf is not None: - if 'monitor' in oldconf: - MonitorConf = """ -[monitor] -ServerMonitorInterval = %s -NetworkMonitorInterval = %s -DiskMonitorInterval = %s -ServiceMonitorInterval = %s -""" % (oldconf['monitor']['ServerMonitorInterval'], oldconf['monitor']['NetworkMonitorInterval'], - oldconf['monitor']['DiskMonitorInterval'], oldconf['monitor']['ServiceMonitorInterval']) - else: - MonitorConf = """ -[monitor] -ServerMonitorInterval = 5000 -NetworkMonitorInterval = 5000 -DiskMonitorInterval = 5000 -ServiceMonitorInterval = 5000 - """ - else: - MonitorConf = """ -[monitor] -ServerMonitorInterval = 5000 -NetworkMonitorInterval = 5000 -DiskMonitorInterval = 5000 -ServiceMonitorInterval = 5000 - """ - - Conf += MonitorConf - - - if not os.path.exists(KsanEtcPath): - os.makedirs(KsanEtcPath) - - with open(MonServicedConfPath, 'w') as f: - f.write(Conf) - return True, '' - -def RegisterNode(): - Cmd = '%s register' % KsanAgentPath - out, err = shcall(Cmd) - if 'Done' in out or 'already registered' in out: - return True, '' - else: - return False, err - - -def StartksanAgent(): - Cmd = '%s stop;%s start -i %s' % (KsanAgentPath, KsanAgentPath, LocalIp) - out, err = shcall(Cmd) - if 'Done' in out or 'already running': - return True, '' - else: - return False, err - -def StartksanMon(): - Cmd = 'kill -9 $(ps -ef |grep ksanMonitor|grep -v grep|awk \'{print $2}\') >/dev/null 2>&1;sleep 1;/usr/local/ksan/bin/ksanMonitor start' - out, err = shcall(Cmd) - if 'Done' in out or 'already running': - return True, '' - else: - return False, err - - - -if __name__ == '__main__': - usage = "Usage: %prog [option]" - parser = MyOptionParser(usage=usage) - #parser.add_option('-S', "--Server", action='store_true', - # dest='Server', default=False, help='Server Option') - parser.add_option('-i', "--local-ip", dest="localip", help='local ip') - parser.add_option('-m', "--portal-host", dest="portalhost", help='portal ip') - parser.add_option('-r', "--rabbitmq-host", dest="mqhost", help='portal ip') - parser.add_option('-p', "--portal-port", dest="portalport", help='portal port') - parser.add_option('-q', "--mq-port", dest="mqport", help='mq port') - parser.add_option('-u', "--mq-user", dest="mquser", help='mq user') - parser.add_option('-w', "--mq-password", dest="mqpassword", help='mq password') - parser.add_option('-k', "--apikey", dest="apikey", help='portal api key') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='Debug mode') - - options, args = parser.parse_args() - PortalIp = options.portalhost - PortalPort = options.portalport - LocalIp = options.localip - MqPort = options.mqport - MqUser = options.mquser - MqHost = options.mqhost - MqPassword = options.mqpassword - ApiKey = options.apikey - - - - #logger = None - #if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - - if not(LocalIp and PortalIp and PortalPort and MqPort and MqUser and MqPassword and ApiKey): - logger.error('Option is omitted LocalIp:%s, PortalIp:%s, PortalPort:%s, MqUser:%s MqPort:%s MqPassword:%s ApiKey:%s' - % (LocalIp, PortalIp, PortalPort, MqUser, MqPort, MqPassword, ApiKey)) - parser.print_help() - sys.exit(-1) - else: - logger.debug('ksanNodeRegister options LocalIp:%s, PortalHost:%s, PortalPort:%s, MqHost:%s, MqUser:%s MqPort:%s MqPassword:%s ApiKey:%s' - % (LocalIp, PortalIp, PortalPort, MqHost, MqUser, MqPort, MqPassword, ApiKey)) - - ret = DependencyCheck() - if ret is False: - logger.error('dependency fail') - sys.exit(-1) - - ret, err = UpdateKsanAgentConfig(PortalIp, PortalPort, LocalIp, MqHost, MqUser, MqPort, MqPassword, ApiKey) - if ret is False: - print(err) - logger.error('fail to update ksanMonitor.conf %s ' % str(err)) - sys.exit(-1) - - if ret is True or 'already registered' not in err: - ret, err = RegisterNode() - if ret is False: - print(err) - logger.error('fail to register ksan server %s ' % str(err)) - - ret, err = StartksanAgent() - if ret is False: - print(err) - logger.error('fail to start ksanAgent service %s ' % str(err)) - sys.exit(-1) - - print('Done') - logger.info('success to register ksan node') diff --git a/core/common/mqmanage/RabbitMqReceiver.py b/core/common/mqmanage/RabbitMqReceiver.py index fc82207f..66119da3 100644 --- a/core/common/mqmanage/RabbitMqReceiver.py +++ b/core/common/mqmanage/RabbitMqReceiver.py @@ -14,16 +14,10 @@ import os, sys if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from disk.disk_mq_handle import * -from service.service_mq_handle import * -import mqmanage -from Enums.EnumResponseResult import EnumResponseResult -from common.ResponseMqData import ResponseMqData -from const.mq import RoutKeyDisk, RoutKeyDiskRpcFinder, RoutKeyDiskPool, RoutKeyService, RoutKeyServiceRpcFinder, \ - RoutKeyNetwork, RoutKeyNetworkRpcFinder, RoutKeyServerUpdate, RoutKeyServerAdd, RoutKeyServerDel +#from service.service_mq_handle import MqServiceHandler +#import mqmanage from mqmanage.RabbitMqConfiguration import RabbitMqConfiguration -import network.network_mq_handle -from server.server_mq_handle import * +from portal_api.apis import * import json import pika @@ -54,17 +48,27 @@ def __init__(self, config: RabbitMqConfiguration, queueName: str, exchangeName: self.MonConf = MonConf self.GlobalFlag = GlobalFlag # {'ServiceUpdated': Updated, 'DiskUpdated': Updated, 'NetworkUpdated': Checked} # Rabbit MQ 리스너 초기화 - self.initializeRabbitMqListener(self.m_config) + while True: + try: + self.initializeRabbitMqListener(self.m_config) + + #result = self.m_channel.queue_declare(queue='', exclusive=True) + #self.m_callback_queue = result.method.queue + print(self.m_queue_name) + self.m_channel.basic_consume( + queue=self.m_queue_name, + on_message_callback=self.on_response, + auto_ack=False) + + self.m_channel.start_consuming() + except pika.exceptions.AMQPChannelError as err: + logger.error('1rabbitmq server connection fail %s %d' % (str(err), sys.exc_info()[2].tb_lineno)) + time.sleep(5) + continue + except Exception as err: + logger.error('2rabbitmq server connection fail %s %d' % (str(err), sys.exc_info()[2].tb_lineno)) + time.sleep(5) - #result = self.m_channel.queue_declare(queue='', exclusive=True) - #self.m_callback_queue = result.method.queue - print(self.m_queue_name) - self.m_channel.basic_consume( - queue=self.m_queue_name, - on_message_callback=self.on_response, - auto_ack=False) - - self.m_channel.start_consuming() """ Rabbit MQ 리스너 초기화 Args: @@ -174,7 +178,7 @@ def MqReceiverHanlder(self, RoutingKey, body, Response): #Response.IsProcessed = True return ResponseReturn elif RoutKeyNetwork in RoutingKey or RoutKeyNetworkRpcFinder.search(RoutingKey): - ResponseReturn = network.network_mq_handle.MqNetworkHandler(self.MonConf, RoutingKey, body, Response, self.ServerId, self.ServiceList, self.GlobalFlag, self.logger) + ResponseReturn = MqNetworkHandler(self.MonConf, RoutingKey, body, Response, self.ServerId, self.ServiceList, self.GlobalFlag, self.logger) Response.IsProcessed = True return ResponseReturn elif RoutKeyServerUpdate in RoutingKey or RoutKeyServerDel in RoutingKey or RoutKeyServerAdd in RoutingKey: @@ -184,7 +188,7 @@ def MqReceiverHanlder(self, RoutingKey, body, Response): else: self.logger.debug("Skip RouteKey: %s" % str(RoutingKey)) Response.IsProcessed = True - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) + ResponseReturn = MqReturn(ResultSuccess) return ResponseReturn diff --git a/core/common/mqmanage/RabbitMqRpc.py b/core/common/mqmanage/RabbitMqRpc.py index 62396f1e..bd745283 100644 --- a/core/common/mqmanage/RabbitMqRpc.py +++ b/core/common/mqmanage/RabbitMqRpc.py @@ -16,9 +16,10 @@ import pika import json sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) from Enums import EnumResponseResult -from common.ResponseData import ResponseData -from common.ResponseDataWithData import ResponseDataWithData +from const.mq import * from mqmanage import RabbitMqConfiguration diff --git a/core/common/mqmanage/RabbitMqSender.py b/core/common/mqmanage/RabbitMqSender.py index 3a6a4db2..8d0531ce 100644 --- a/core/common/mqmanage/RabbitMqSender.py +++ b/core/common/mqmanage/RabbitMqSender.py @@ -11,94 +11,108 @@ """ import os, sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -import pika -import json -from Enums.EnumResponseResult import EnumResponseResult -from common.ResponseData import ResponseData +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from const.mq import * from mqmanage.RabbitMqConfiguration import RabbitMqConfiguration class RabbitMqSender: - """ 생성자 - Args: - config: Rabbit MQ 설정 객체 - Returns: - 없음 - """ - def __init__(self, config: RabbitMqConfiguration): - self.m_connection = None - self.m_channel = None - self.m_config = config - - # Rabbit MQ 리스너 초기화 - self.initializeRabbitMqListener(self.m_config) - - """ 소멸자 - Args: - 없음 - Returns: - 없음 - """ - def __del__(self): - # 채널 연결이 되어 있는 경우, 채널 연결 해제 - if self.m_channel is not None: - self.m_channel.close() - self.m_channel = None - - # 연결이 되어 있는 경우, 연결 해제 - if self.m_connection is not None: - self.m_connection.close() - self.m_connection = None - - """ Rabbit MQ 리스너 초기화 - Args: - config: Rabbit MQ 설정 객체 - Returns: - 없음 - """ - def initializeRabbitMqListener(self, config: RabbitMqConfiguration): - - # 계정 정보 생성 - credentials = pika.PlainCredentials(username=config.user, password=config.password) - - # 연결 파라미터 생성 - connectionParams = pika.ConnectionParameters( - host=config.host - , port=config.port - , virtual_host=config.virtualHost - , credentials=credentials - ) - - # 연결 객체 생성 - self.m_connection = pika.BlockingConnection(connectionParams) - # 채널 생성 - self.m_channel = self.m_connection.channel() - - """ 객체를 Rabbit MQ로 전송한다. - Args: - exchange: Exchange 명 - routingKey: 라우팅 키 - sendingObject: 전송할 객체 - Returns: - 전송 결과 응답 객체 - """ - def send(self, exchange, routingKey, sendingObject): - - # 전송할 객체가 존재하지 않는 경우, 에러 반환 - if sendingObject is None: - return ResponseData(EnumResponseResult.Error, "MQ001", "Invalid Request") - - # 전송할 데이터로 변환 - message = json.dumps(sendingObject) - - # 메시지 전송 - self.m_channel.basic_publish( - exchange=exchange - , routing_key=routingKey - , body=message) - - print("[Rabbit MQ] Data transfer was successful. (exchange: %r, routingKey: %r, message: %r)" % (exchange, routingKey, message)) - - return ResponseData(EnumResponseResult.Success, "", "") + """ 생성자 + Args: + config: Rabbit MQ 설정 객체 + Returns: + 없음 + """ + def __init__(self, config: RabbitMqConfiguration, logger=None): + self.m_connection = None + self.m_channel = None + self.m_config = config + self.logger = logger + + # Rabbit MQ 리스너 초기화 + + self.initializeRabbitMqListener(self.m_config) + """ + while True: + try: + self.initializeRabbitMqListener(self.m_config) + except Exception as err: + print('$$$$$ fail error %s' % str(err)) + time.sleep(5) + """ + + + """ 소멸자 + Args: + 없음 + Returns: + 없음 + """ + def __del__(self): + # 채널 연결이 되어 있는 경우, 채널 연결 해제 + if self.m_channel is not None: + self.m_channel.close() + self.m_channel = None + + # 연결이 되어 있는 경우, 연결 해제 + if self.m_connection is not None: + self.m_connection.close() + self.m_connection = None + + """ Rabbit MQ 리스너 초기화 + Args: + config: Rabbit MQ 설정 객체 + Returns: + 없음 + """ + def initializeRabbitMqListener(self, config: RabbitMqConfiguration): + + # 계정 정보 생성 + credentials = pika.PlainCredentials(username=config.user, password=config.password) + + # 연결 파라미터 생성 + connectionParams = pika.ConnectionParameters( + host=config.host + , port=config.port + , virtual_host=config.virtualHost + , credentials=credentials + ) + + try: + # 연결 객체 생성 + self.m_connection = pika.BlockingConnection(connectionParams) + # 채널 생성 + self.m_channel = self.m_connection.channel() + except Exception as err: + self.logger.error('fail to connect to rabbitmq server %s' % str(err)) + time.sleep(5) + + """ 객체를 Rabbit MQ로 전송한다. + Args: + exchange: Exchange 명 + routingKey: 라우팅 키 + sendingObject: 전송할 객체 + Returns: + 전송 결과 응답 객체 + """ + def send(self, exchange, routingKey, sendingObject): + try: + # 전송할 객체가 존재하지 않는 경우, 에러 반환 + if sendingObject is None: + return ResponseData(EnumResponseResult.Error, "MQ001", "Invalid Request") + # 전송할 데이터로 변환 + message = json.dumps(sendingObject) + + # 메시지 전송 + self.m_channel.basic_publish( + exchange=exchange + , routing_key=routingKey + , body=message) + + self.logger.debug("[Rabbit MQ] Data transfer was successful. (exchange: %r, routingKey: %r, message: %r)" % (exchange, routingKey, message)) + except Exception as err: + self.logger.error('fail to send to rabbitmq %s' % str(err)) + + return ResponseData(EnumResponseResult.Success, "", "") diff --git a/core/common/mqmanage/mq.py b/core/common/mqmanage/mq.py index 5c958c8c..8897da76 100644 --- a/core/common/mqmanage/mq.py +++ b/core/common/mqmanage/mq.py @@ -18,12 +18,10 @@ import pdb if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from const.common import * from mqmanage.RabbitMqReceiver import RabbitMqReceiver from mqmanage.RabbitMqConfiguration import RabbitMqConfiguration from mqmanage.RabbitMqRpc import RabbitMqRpc from mqmanage.RabbitMqSender import RabbitMqSender -import json class Mq: def __init__(self, Host, Port, VirtualHost, User, Password, RoutingKey, ExchangeName, QueueName=None, ServerId=None, @@ -45,7 +43,7 @@ def __init__(self, Host, Port, VirtualHost, User, Password, RoutingKey, Exchange self.GlobalFlag = GlobalFlag def Sender(self, Data): - mqSender = RabbitMqSender(self.config) + mqSender = RabbitMqSender(self.config, logger=self.logger) mqSender.send(self.config.exchangeName, self.routingkey, Data) def Receiver(self): @@ -55,26 +53,3 @@ def Receiver(self): def Rpc(self, Data): mqRpc = RabbitMqRpc(self.config) mqRpc.send(self.config.exchangeName, self.config.exchangeName, Data) - - -@catch_exceptions() -def MqReturn(Result, Code=0, Messages='', Data=None): - Ret = dict() - if Result is True or Result == ResultSuccess: - Ret['Result'] = 'Success' - else: - Ret['Result'] = 'Error' - Ret['Code'] = Code - - Ret['Message'] = Messages - if Data: - Ret['Data'] = Data - #Ret['Data'] = json.dumps(Ret) - return json.dumps(Ret) - -def RemoveQueue(QueueHost, QueueName): - connection = pika.BlockingConnection(pika.ConnectionParameters(QueueHost)) - channel = connection.channel() - - channel.queue_delete(queue=QueueName) - connection.close() \ No newline at end of file diff --git a/core/common/network/network_manage.py b/core/common/network/network_api.py similarity index 88% rename from core/common/network/network_manage.py rename to core/common/network/network_api.py index fb3500e4..345adfd7 100644 --- a/core/common/network/network_manage.py +++ b/core/common/network/network_api.py @@ -12,13 +12,20 @@ import os import sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.network import GetNetwork +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from const.common import * +#from common.network import GetNetwork +#from common.httpapi import RestApi from const.network import ResPonseVlanNetworkInterfaceItems, RequestNetworkInterfaceItems, \ - RequestVlanNetworkInterfaceInfo, NetworkInterfaceLinkStateItems, RequestNetworkInterfaceCheck, RequestVlanNetworkInterfaceCheck -from server.server_manage import * -from const.network import NetworkInterfaceItemsModule -from const.http import ResponseHeaderModule + RequestVlanNetworkInterfaceInfo, NetworkInterfaceLinkStateItems, RequestNetworkInterfaceCheck, \ + RequestVlanNetworkInterfaceCheck +from const.common import NetworkInterfaceItemsModule +#from const.common import * +#from const.http import ResPonseHeader, ResponseHeaderModule +#from server.server_api import GetServerInfo, GetAllServerDetailInfo +from portal_api.apis import * + def ManageNetworkInterface(): @@ -82,14 +89,14 @@ def GetAllNetworkInterface(Ip, Port, ServerId=None, logger=None): @catch_exceptions() -def GetNetworkInterfaceDetailWithId(Ip, Port, InterfaceId, logger=None): +def GetNetworkInterfaceDetailWithId(PortalIp, PortalPort, PortalApiKey, InterfaceId, logger=None): """ Get Server Detail info with Interface Id :param InterfaceId: :param logger: :return:If Success, ServerItemsDetailModule ojbect, otherwise None """ - Res, Errmsg, Ret, Data = GetAllServerDetailInfo(Ip, Port, logger=logger) + Res, Errmsg, Ret, Data = GetAllServerDetailInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) if Res == ResOk: if Ret.Result == ResultSuccess: for Svr in Data: @@ -432,6 +439,49 @@ def check_exist_vlan_network_interface(ip, port, ServerId, NicId, Tag, VlanId=No return Ret, Errmsg, Header.Data else: return Ret, Errmsg, None + + + + + +@catch_exceptions() +def MqNetworkHandler(MonConf, RoutingKey, Body, Response, ServerId, ServiceList, GlobalFlag, logger): + logger.debug("%s %s %s" % (str(RoutingKey), str(Body), str(Response))) + ResponseReturn = MqReturn(ResultSuccess) + Body = Body.decode('utf-8') + Body = json.loads(Body) + body = DictToObject(Body) + if RoutKeyNetworkAddFinder.search(RoutingKey) or RoutKeyNetworkUpdateFinder.search(RoutingKey): + GlobalFlag['NetworkUpdated'] = Updated + if body.ServerId == ServerId: + Response.IsProcessed = True + ret, errmsg = ManageNetworkInterface() + if ret is False: + ResponseReturn = MqReturn(ret, Code=1, Messages='fail') + print(ResponseReturn) + return ResponseReturn + + elif RoutKeyNetworkAddedFinder.search(RoutingKey): + ServerId = body.ServerId + IpAddress = body.IpAddress + Res, Errmsg , Ret, Data = GetServerInfo(MonConf.PortalHost, int(MonConf.PortalPort),MonConf.PortalApiKey, ServerId=ServerId, logger=logger) + if Res == ResOk: + if Ret.Result == ResultSuccess: + HostName = Data.Name + HostInfo = [(IpAddress, HostName)] + UpdateEtcHosts(HostInfo, 'add') + logging.log(logging.INFO, 'new host is added. %s' % str(HostInfo)) + else: + logger.error('fail to add hostname to /etc/hosts with ip: %s %s' % (IpAddress, Ret.Message)) + else: + logger.error('fail to add hostname to /etc/hosts with ip: %s %s' % (IpAddress, Errmsg)) + + + + else: + Response.IsProcessed = True + return ResponseReturn +''' class MyOptionParser(OptionParser): def print_help(self): Usage = """ @@ -453,3 +503,4 @@ def print_help(self): print(Usage) """ +''' \ No newline at end of file diff --git a/core/common/network/network_mq_handle.py b/core/common/network/network_mq_handle.py index 14b56df4..4b57cafb 100644 --- a/core/common/network/network_mq_handle.py +++ b/core/common/network/network_mq_handle.py @@ -14,18 +14,18 @@ import os, sys import pdb import time -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -import mqmanage.mq -from common.shcommand import UpdateEtcHosts -from common.init import WaitAgentConfComplete, GetAgentConfig +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) + +#from const.common import * from const.network import RequestNetworkInterfaceStat, NetworkInterfaceLinkStateItems -from network.network_manage import * -from const.mq import RoutKeyNetworkUsage, ExchangeName, RoutKeyNetworkLinkState, RoutKeyNetworkAddFinder, \ - RoutKeyNetworkUpdateFinder, RoutKeyNetworkAddedFinder +#from const.mq import RoutKeyNetworkUsage, ExchangeName, RoutKeyNetworkLinkState, RoutKeyNetworkAddFinder, \ +# RoutKeyNetworkUpdateFinder, RoutKeyNetworkAddedFinder +from common.utils import * +from const.common import * from mqmanage.mq import * -import jsonpickle -import logging -import inspect +from portal_api.apis import * + @catch_exceptions() def UpdateNetworkStat(Conf, GlobalFlag, logger): @@ -44,8 +44,8 @@ def UpdateNetworkStat(Conf, GlobalFlag, logger): #mqSender.send(config.exchangeName, "*", data) LocalDev = GetNetwork() - NetworkUsageMq = Mq(conf.MQHost, int(conf.MQPort), '/', conf.MQUser, conf.MQPassword, RoutKeyNetworkUsage, ExchangeName) - NetworkLinkStateMq = Mq(conf.MQHost, int(conf.MQPort), '/', conf.MQUser, conf.MQPassword, RoutKeyNetworkLinkState, ExchangeName) + NetworkUsageMq = Mq(conf.MQHost, int(conf.MQPort), '/', conf.MQUser, conf.MQPassword, RoutKeyNetworkUsage, ExchangeName, logger=logger) + NetworkLinkStateMq = Mq(conf.MQHost, int(conf.MQPort), '/', conf.MQUser, conf.MQPassword, RoutKeyNetworkLinkState, ExchangeName, logger=logger) #Res, Errmsg, Ret, AllNetDevs = GetNetworkInterface(conf.mgs.MgsIp, int(conf.mgs.IfsPortalPort), conf.mgs.ServerId) NetworkMonitorInterval = int(conf.NetworkMonitorInterval)/1000 while True: @@ -89,40 +89,3 @@ def UpdateNetworkStat(Conf, GlobalFlag, logger): time.sleep(IntervalMiddle) -@catch_exceptions() -def MqNetworkHandler(MonConf, RoutingKey, Body, Response, ServerId, ServiceList, GlobalFlag, logger): - logger.debug("%s %s %s" % (str(RoutingKey), str(Body), str(Response))) - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Body = Body.decode('utf-8') - Body = json.loads(Body) - body = DictToObject(Body) - if RoutKeyNetworkAddFinder.search(RoutingKey) or RoutKeyNetworkUpdateFinder.search(RoutingKey): - GlobalFlag['NetworkUpdated'] = Updated - if body.ServerId == ServerId: - Response.IsProcessed = True - ret, errmsg = ManageNetworkInterface() - if ret is False: - ResponseReturn = mqmanage.mq.MqReturn(ret, Code=1, Messages='fail') - print(ResponseReturn) - return ResponseReturn - - elif RoutKeyNetworkAddedFinder.search(RoutingKey): - ServerId = body.ServerId - IpAddress = body.IpAddress - Res, Errmsg , Ret, Data = GetServerInfo(MonConf.PortalHost, int(MonConf.PortalPort),MonConf.PortalApiKey, ServerId=ServerId, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - HostName = Data.Name - HostInfo = [(IpAddress, HostName)] - UpdateEtcHosts(HostInfo, 'add') - logging.log(logging.INFO, 'new host is added. %s' % str(HostInfo)) - else: - logger.error('fail to add hostname to /etc/hosts with ip: %s %s' % (IpAddress, Ret.Message)) - else: - logger.error('fail to add hostname to /etc/hosts with ip: %s %s' % (IpAddress, Errmsg)) - - - - else: - Response.IsProcessed = True - return ResponseReturn diff --git a/portal/Portal/src/app/errors/403/error-403-page.component.ts b/core/common/portal_api/apis.py similarity index 64% rename from portal/Portal/src/app/errors/403/error-403-page.component.ts rename to core/common/portal_api/apis.py index 262fe2c3..06d8a234 100644 --- a/portal/Portal/src/app/errors/403/error-403-page.component.ts +++ b/core/common/portal_api/apis.py @@ -1,4 +1,5 @@ -/* +#!/usr/bin/env python3 +""" * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version @@ -7,20 +8,17 @@ * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -*/ -import {Component, OnInit} from '@angular/core'; +""" -@Component({ - selector: 'app-error-403-page', - templateUrl: './error-403-page.component.html', - styleUrls: ['./error-403-page.component.scss'] -}) -export class Error403PageComponent implements OnInit { - constructor() { - } +import os +import sys +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) - ngOnInit(): void { - } - -} +from server.server_api import * +from network.network_api import * +from service.service_api import * +from disk.disk_api import * +from user.user_manage import * +from disk.diskpool_manage import * \ No newline at end of file diff --git a/core/common/rest/agent_api_server.py b/core/common/rest/agent_api_server.py new file mode 100644 index 00000000..c5748c57 --- /dev/null +++ b/core/common/rest/agent_api_server.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +""" +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +""" +import os, sys +from fastapi import FastAPI +import uvicorn + +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) + +from const.rest import RestReturn +from const.common import AgentConf +from server.server_manage import RestHandlerServer + + +def RestServer(port): + app = FastAPI() + + """ + @app.get("/") + async def read_root(): + return {"Hello": "World"} + + @app.get("/items/{item_id}") + async def read_item(item_id: int, q: Union[str, None] = None): + return RestReturn + """ + + @app.post(path="/api/v1/Servers/Initialize", description="시스템 접속 정보를 초기화한다.", response_model=RestReturn) + async def ServerInitialize(conf: AgentConf): + rest = RestHandlerServer() + return rest.InitAgentConf(conf) + + @app.post("/api/v1/Servers") + #async def ServerAdd(LocalIp: str, PortalHost: str, PortalPort: int, MQHost: str, MQPort: int, MQPassword: str, + # MQUser: str, PortalApikey: str): + async def ServerAdd(conf: AgentConf): + rest = RestHandlerServer() + return rest.AddServer(conf) + + return app + +app = FastAPI() + +""" +@app.get("/") +async def read_root(): + return {"Hello": "World"} + +@app.get("/items/{item_id}") +async def read_item(item_id: int, q: Union[str, None] = None): + return RestReturn +""" + +@app.post(path="/api/v1/Servers/Initialize", description="시스템 접속 정보를 초기화한다.", response_model=RestReturn) +async def ServerInitialize(conf: AgentConf): + rest = RestHandlerServer() + return rest.InitAgentConf(conf) + +@app.post(path="/api/v1/Servers", description="시스템 접속 정보를 초기화 하고 해당 서버를 시스템에 등록한다.", response_model=RestReturn) +#async def ServerAdd(LocalIp: str, PortalHost: str, PortalPort: int, MQHost: str, MQPort: int, MQPassword: str, +# MQUser: str, PortalApikey: str): +async def ServerAdd(conf: AgentConf): + rest = RestHandlerServer() + return rest.AddServer(conf) + + +def StartRestServer(Port, AgentConf, logger=None): + #App = RestServer(Port) + + uvicorn.run(app, host='0.0.0.0', port=Port) \ No newline at end of file diff --git a/core/common/server/server_api.py b/core/common/server/server_api.py new file mode 100644 index 00000000..4650e3c9 --- /dev/null +++ b/core/common/server/server_api.py @@ -0,0 +1,492 @@ +#!/bin/env python3 +""" +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +""" + +import os, sys +import psutil +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +#from common.httpapi import * +#from const.common import * +from const.mq import * +from const.server import ServerItemsDetailModule, ServerItemsModule, RequestServerInfo, RequestServerInitInfo, \ + UpdateServerInfoItems, RequestServerExistCheck +from common.utils import * +from common.base_utils import * + + +""" +############## Servers ############### +""" + + +class GetServerUsage(object): + def __init__(self, ServerId): + self.ServerId = ServerId + self.LoadAverage1M = None + self.LoadAverage5M = None + self.LoadAverage15M = None + self.MemoryTotal = 0 + self.MemoryUsed = 0 + + def Get(self): + load = psutil.getloadavg() + self.LoadAverage1M = load[0] + self.LoadAverage5M = load[1] + self.LoadAverage15M = load[2] + Mem = psutil.virtual_memory() + self.MemoryTotal = Mem.total + self.MemoryUsed = Mem.used + + +@catch_exceptions() +def GetServerInfo(Ip, Port, ApiKey, ServerId=None, Name=None, logger=None): + """ + get server info all or specific server info with Id + :param ip: + :param port: + :param disp: + :return: if Id is None, ServerItemsDetail object is returned. otherwise ServerItems list returned + """ + + if ServerId is not None: + TargetServer = ServerId + elif Name is not None: + TargetServer = Name + else: + TargetServer = None + + ItemsHeader = True + if TargetServer is not None: + Url = "/api/v1/Servers/%s" % TargetServer + ItemsHeader = False + ReturnType = ServerItemsDetailModule + else: + Url = "/api/v1/Servers" + ReturnType = ServerItemsModule + Params = dict() + Params['countPerPage'] = 100 + Conn = RestApi(Ip, Port, Url, authkey=ApiKey, params=Params, logger=logger) + Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) + if Res == ResOk: + AllServerInfo = list() + if Ret.Result == ResultSuccess: + if TargetServer is not None: + return Res, Errmsg, Ret, Ret.Data + else: + return Res, Errmsg, Ret, Ret.Data.Items + else: + return Res, Errmsg, Ret, AllServerInfo + else: + return Res, Errmsg, None, None + + +def GetAllServerDetailInfo(Ip, Port, Apikey, logger=None): + Res, Errmsg, Ret, Servers = GetServerInfo(Ip, Port, Apikey, logger=logger) + if Res == ResOk: + AllServerDetailInfo = list() + if Ret.Result == ResultSuccess: + for Svr in Servers: + Res, Errmsg, Ret, Detail = GetServerInfo(Ip, Port, Apikey, ServerId=Svr.Id, logger=logger) + if Res == ResOk: + AllServerDetailInfo.append(Detail) + else: + print(Errmsg) + return Res, Errmsg, Ret, AllServerDetailInfo + else: + print('fail to get Server List', Ret.Message) + return Res, Errmsg, Ret, None + else: + print('fail to get Server List', Errmsg) + return Res, Errmsg, Ret, None + + +@catch_exceptions() +def ShowServerInfo(Data, Id=None, Detail=False, SysinfoDisp=False): + + Data = ServerListOrdering(Data) + + if SysinfoDisp is True: + ServerTitleLine = '%s' % ('=' * 105) + ServerDataLine = '%s' % ('-' * 105) + title ="|%s|%s|%s|%s|" % ('ServerName'.ljust(30), 'IpAddress'.ljust(30), 'Status'.ljust(10), ' ' * 30) + else: + + if Detail == MoreDetailInfo: + ServerTitleLine = '%s' % ('=' * 136) + ServerDataLine = '%s' % ('-' * 136) + title ="|%s|%s|%s|%s|%s|%s|%s|" % ('ServerName'.ljust(20), 'IpAddress'.ljust(15), 'Status'.ljust(10), + 'LoadAvg 1M 5M 15M'.ljust(23), 'MemTotal'.ljust(11), + 'MemUsed'.ljust(11), 'Id'.ljust(38)) + elif Detail == DetailInfo: + ServerTitleLine = '%s' % ('=' * 97) + ServerDataLine = '%s' % ('-' * 97) + title ="|%s|%s|%s|%s|%s|%s|" % ('ServerName'.ljust(20), 'IpAddress'.ljust(15), 'Status'.ljust(10), + 'LoadAvg 1M 5M 15M'.ljust(23), 'MemTotal'.ljust(11), 'MemUsed'.ljust(11)) + else: + ServerTitleLine = '%s' % ('=' * 49) + ServerDataLine = '%s' % ('-' * 49) + title ="|%s|%s|%s|" % ('ServerName'.ljust(20), 'IpAddress'.ljust(15), 'Status'.ljust(10)) + + print(ServerTitleLine) + print(title) + print(ServerTitleLine) + if Id is None: + for svr in Data: + if len(svr.NetworkInterfaces) > 0: + ManagementIp = svr.NetworkInterfaces[0].IpAddress + else: + ManagementIp = '-' + + if SysinfoDisp is True: + _svr = "|%s|%s|%s|%s|" % (svr.Name.ljust(30), ManagementIp.ljust(30), DisplayState(svr.State).ljust(10), ' ' * 30) + else: + if Detail == MoreDetailInfo: + _svr ="|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % \ + (svr.Name.ljust(20), ManagementIp.ljust(15), + DisplayState(svr.State).ljust(10), "{:,}".format(svr.LoadAverage1M).rjust(7), "{:,}".format(svr.LoadAverage5M).rjust(7) , + "{:,}".format(svr.LoadAverage15M).rjust(7), str(Byte2HumanValue(int(svr.MemoryTotal), 'TotalSize')).rjust(20), + str(Byte2HumanValue(int(svr.MemoryUsed), 'UsedSize')).rjust(20), svr.Id.ljust(38)) + + elif Detail == DetailInfo: + _svr ="|%s|%s|%s|%s|%s|%s|%s|%s|" % \ + (svr.Name.ljust(20), ManagementIp.ljust(15), + DisplayState(svr.State).ljust(10), "{:,}".format(svr.LoadAverage1M).rjust(7), "{:,}".format(svr.LoadAverage5M).rjust(7) , + "{:,}".format(svr.LoadAverage15M).rjust(7), str(Byte2HumanValue(int(svr.MemoryTotal), 'TotalSize')).rjust(20), + str(Byte2HumanValue(int(svr.MemoryUsed), 'UsedSize')).rjust(20)) + + else: + _svr = "|%s|%s|%s|" % \ + (svr.Name.ljust(20), ManagementIp.ljust(15), DisplayState(svr.State).ljust(10)) + + print(_svr) + print(ServerDataLine) + else: + svr = Data[0] + if Detail: + _svr = "|%s|%s|%s|%s|%s|%s|" % \ + (svr.Name.ljust(20), '{:30.30}'.format(svr.Description.ljust(30)), + '{:30.30}'.format(svr.CpuModel.ljust(30)), str(svr.Clock).ljust(20), + svr.State.ljust(20), svr.Id.ljust(30)) + else: + _svr = "|%s|%s|%s|" % (svr.Name.ljust(20), svr.State.ljust(20), svr.Id.ljust(30)) + print(_svr) + + +def ServerListOrdering(ServerList): + NewServerList = list() + ServerNameDict = dict() + for serverinfo in ServerList: + ServerName = serverinfo.Name + ServerNameDict[ServerName] = serverinfo + + for servername in sorted(ServerNameDict.keys(), key=str.casefold): + serverinfo = ServerNameDict[servername] + NewServerList.append(serverinfo) + + return NewServerList + + +@catch_exceptions() +def AddServer(ip, port, ApiKey, Description, logger=None): + """ + register server info + :param ip: string + :param port: integer + :param Description: + :param logger: + :return:tuple(error code, error msg, Success to get result:header, fail to get result: None) + """ + server = RequestServerInfo(Description) + body = jsonpickle.encode(server, make_refs=False) + Url = '/api/v1/Servers' + ReturnType = ResponseHeaderModule + Params = body + Conn = RestApi(ip, port, Url, authkey=ApiKey, params=Params, logger=logger) + Res, Errmsg, Ret = Conn.post(ReturnType=ReturnType) + return Res, Errmsg, Ret + + + +@catch_exceptions() +def ServerInit(PortalIp, PortalPort, PortalApiKey, TargetServerIp, logger=None): + """ + register server info + :param ip: string + :param port: integer + :param Description: + :param logger: + :return:tuple(error code, error msg, Success to get result:header, fail to get result: None) + """ + """ + SshUser = get_input('Insert ssh user', str, default='root') + SshPassword = get_input('Insert ssh password', 'pwd', default='') + if PortalIp is None: # if ksanMonitor.conf is not configured + PortalIp = get_input('Insert portal host', str, default='') + PortalPort = get_input('Insert portal port', int, default='') + PortalApiKey = get_input('Insert portal api key', str, default='') + + if not IsIpValid(PortalIp): + Ret, Hostname, Ip = GetHostInfo(hostname=PortalIp) + if Ret is False: + print('Invalid Hostname') + sys.exit(-1) + else: + PortalIp = Ip + """ + SshUser='' + SshPassword = '' + server = RequestServerInitInfo() + server.Set(TargetServerIp, PortalIp, PortalPort, SshUser, SshPassword) + body = jsonpickle.encode(server, make_refs=False) + Url = '/api/v1/Servers/Initialize' + ReturnType = ResponseHeaderModule + Params = body + Conn = RestApi(PortalIp, PortalPort, Url, authkey=PortalApiKey, params=Params, logger=logger) + Res, Errmsg, Ret = Conn.post(ReturnType=ReturnType) + return Res, Errmsg, Ret + + +@catch_exceptions() +def RegisterServer(ip, port, ApiKey, Description, Name=None, logger=None): + """ + register server info to server pool + :param ip: string + :param port: integer + :param Description: + :param Name : In case of already exists, Search the server info with Name(hostname) + :param logger: + :return:if success or alredy registered, return tuple(error code, error msg, Header, ServerItems object), otherwise tuple(Res, Errmsg, None, None) + """ + server = RequestServerInfo(Description) + body = jsonpickle.encode(server, make_refs=False) + Url = '/api/v1/Servers' + ItemsHeader = False + ReturnType = ServerItemsDetailModule + Params = body + Conn = RestApi(ip, port, Url, authkey=ApiKey, params=Params, logger=logger) + + Res, Errmsg, Ret = Conn.post(ItemsHeader=ItemsHeader, ReturnType=ReturnType) + if Res == ResOk: + if Ret.Result == ResultSuccess: + return Res, Errmsg, Ret, Ret.Data + elif Ret.Code == CodeDuplicated: + res, errmsg, ret, AllServerInfo = GetServerInfo(ip, port, ApiKey, logger=logger) + if res == ResOk: + for svr in ret.Data.Items: + if svr.Name == Name: + return Res, Errmsg, Ret, svr + + return Res, Errmsg, Ret, None + + +@catch_exceptions() +def UpdateServerInfo(Ip, Port, Authkey, ServerId=None, Name=None, Description=None, State=None, logger=None): + """ + update server info + :param ip: + :param port: + :param Description: + :param Id: + :param logger: + """ + if ServerId is not None: + TargetServer = ServerId + elif Name is not None: + TargetServer = Name + else: + return ResInvalidCode, ResInvalidMsg, None + + Res, Errmsg, Ret, Data = GetServerInfo(Ip, Port, Authkey, ServerId=TargetServer, logger=logger) + if Res == ResOk: + if Ret.Result == ResultSuccess: + if Name is not None: + Data.Name = Name + if Description is not None: + Data.Description = Description + if State is not None: + Data.State = State + + Url = '/api/v1/Servers/%s' % TargetServer + server = UpdateServerInfoItems() + server.Set(Data.Name, Data.Description, Data.CpuModel, Data.Clock, Data.State, Data.Rack, Data.MemoryTotal) + body = jsonpickle.encode(server, make_refs=False) + Params = body + Conn = RestApi(Ip, Port, Url, authkey=Authkey, params=Params, logger=logger) + Ret, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResPonseHeader) + return Ret, Errmsg, Data + else: + return Res, Errmsg, Ret + else: + return Res, Errmsg, None + + + + + +@catch_exceptions() +def UpdateServerUsage(Ip, Port, ServerId, LoadAverage1M=None, LoadAverage5M=None, LoadAverage15M=None, MemoryUsed=None, State=None, logger=None): + """ + update server info + :param ip: + :param port: + :param Description: + :param Id: + :param logger: + """ + Res, Errmsg, Ret, Data = GetServerInfo(Ip, Port, ServerId=ServerId, logger=logger) + if Res == ResOk: + if Ret.Result == ResultSuccess: + if LoadAverage1M is not None: + Data.LoadAverage1M = LoadAverage1M + if LoadAverage5M is not None: + Data.LoadAverage5M = LoadAverage5M + if LoadAverage15M is not None: + Data.LoadAverage15M = LoadAverage15M + if MemoryUsed is not None: + Data.MemoryUsed = MemoryUsed + + if State is not None: + Data.State = State + + Url = '/api/v1/Servers/%s' % ServerId + server = UpdateServerInfoItems() + server.Set(Data.Name, Data.Description, Data.CpuModel, Data.Clock, Data.State, Data.Rack, Data.MemoryTotal) + body = jsonpickle.encode(server, make_refs=False) + Params = body + Conn = RestApi(Ip, Port, Url, params=Params, logger=logger) + Ret, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResPonseHeader) + return Ret, Errmsg, Data + else: + return Res, Errmsg, Ret + else: + return Res, Errmsg, None + + +@catch_exceptions() +def UpdateServerState(Ip, Port, ServerId, State, logger=None): + """ + update server info + :param ip: + :param port: + :param Description: + :param Id: + :param logger: + """ + Url = '/api/v1/Servers/%s/State/%s' % (ServerId, State) + body = dict() + Params = body + Conn = RestApi(Ip, Port, Url, params=Params, logger=logger) + Ret, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResPonseHeader) + return Ret, Errmsg, Data + + +@catch_exceptions() +def RemoveServer(ip, port, ApiKey, ServerId=None, Name=None, logger=None): + """ + delete server info from server pool + :param ip: + :param port: + :param Id: + :param logger: + :return:tuple(error code, error msg, ResponseHeader class) + """ + + if ServerId is not None: + TargetServer = ServerId + elif Name is not None: + TargetServer = Name + else: + return ResInvalidCode, ResInvalidMsg + ' Id and Name are all None', None + + Url = '/api/v1/Servers/%s' % TargetServer + + ReturnType = ResponseHeaderModule + Conn = RestApi(ip, port, Url, authkey=ApiKey, logger=logger) + Res, Errmsg, Ret = Conn.delete(ReturnType=ReturnType) + return Res, Errmsg, Ret + + +@catch_exceptions() +def server_exist_check_with_servername(ip, port, Name, Authkey, ExcludeServerId=None, logger=None): + """ + check if a specific server exists with Name. + :param ip: + :param port: + :param Name: + :param ExcludeServerId: to exclude the server whose id is ExcludeServerId + :param logger: + :return:tuple(error code, error msg, Success to get result:True/False, fail to get result: None) + """ + server = RequestServerExistCheck(Name) + body = jsonpickle.encode(server, make_refs=False) + if ExcludeServerId is not None: + Url = '/api/v1/Servers/Exist/%s' % ExcludeServerId + else: + Url = '/api/v1/Servers/Exist' + Params = body + Conn = RestApi(ip, port, Url, authkey=Authkey, params=Params, logger=logger) + Res, Errmsg, Ret = Conn.post() + return Res, Errmsg, Ret + + + + +def MqServerHandler(Conf, RoutingKey, Body, Response, ServerId, logger): + logger.debug("MqServerHandler %s %s" % (str(RoutingKey), str(Body))) + try: + ResponseReturn = MqReturn(ResultSuccess) + Body = Body.decode('utf-8') + Body = json.loads(Body) + body = DictToObject(Body) + if RoutKeyServerUpdateFinder.search(RoutingKey) or RoutKeyServerAddFinder.search(RoutingKey): + #IpAddress = body.NetworkInterfaces.IpAddress # not available + pass + + elif RoutKeyServerDelFinder.search(RoutingKey): + HostName = body.Name + HostInfo = [('', HostName)] + UpdateEtcHosts(HostInfo, 'remove') + logging.log(logging.INFO, 'host is removed. %s' % str(HostInfo)) + if ServerId == body.Id: + ret, errlog = RemoveQueue(Conf) + if ret is False: + logging.error('fail to remove queue %s' % errlog) + else: + logging.log(logging.INFO, 'success to remove queue') + if os.path.exists(MonServicedConfPath): + os.unlink(MonServicedConfPath) + logging.log(logging.INFO, 'ksanMonitor.conf is removed') + + ''' + if RoutKeyServerUpdateFinder.search(RoutingKey): + ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) + Body = Body.decode('utf-8') + Body = json.loads(Body) + body = DictToObject(Body) + if ServerId == body.ServerId: + ret = CheckDiskMount(body.Path) + if ret is False: + ResponseReturn = mqmanage.mq.MqReturn(ret, Code=1, Messages='No such disk is found') + Response.IsProcessed = True + print(ResponseReturn) + return ResponseReturn + ''' + return ResponseReturn + except Exception as err: + print(err) + +def RemoveQueue(Conf): + QueueHost = Conf.MQHost + QueueName = 'ksan-agent-%s' % Conf.ServerId + RemoveQueue(QueueHost, QueueName) + return True, '' + diff --git a/core/common/server/server_manage.py b/core/common/server/server_manage.py index 40dd5ec2..43471916 100644 --- a/core/common/server/server_manage.py +++ b/core/common/server/server_manage.py @@ -12,405 +12,20 @@ import os, sys import psutil -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -import jsonpickle -from common.log import Logging -import socket -import logging -from common.httpapi import * -from common.httpapi import RestApi -from common.network import GetNetwork +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from common.utils import RestApi from const.common import * -from common.utils import DisplayState from const.server import ServerItemsDetailModule, ServerItemsModule, RequestServerInfo, RequestServerInitInfo, \ UpdateServerInfoItems, RequestServerExistCheck -from const.http import ResPonseHeader, ResponseHeaderModule -from const.common import AgentConf +from common.utils import ResPonseHeader from const.rest import SetRestReturn -from common.init import get_input, GetConf, UpdateConf -from common.shcommand import GetHostInfo, shcall -from common.utils import IsIpValid -from optparse import OptionParser +from portal_api.apis import * +#from network.network_manage import GetNetworkInterfaceDetailWithId """ ############## Servers ############### """ - -class GetServerUsage(object): - def __init__(self, ServerId): - self.ServerId = ServerId - self.LoadAverage1M = None - self.LoadAverage5M = None - self.LoadAverage15M = None - self.MemoryTotal = 0 - self.MemoryUsed = 0 - - def Get(self): - load = psutil.getloadavg() - self.LoadAverage1M = load[0] - self.LoadAverage5M = load[1] - self.LoadAverage15M = load[2] - Mem = psutil.virtual_memory() - self.MemoryTotal = Mem.total - self.MemoryUsed = Mem.used - - -@catch_exceptions() -def GetServerInfo(Ip, Port, ApiKey, ServerId=None, Name=None, logger=None): - """ - get server info all or specific server info with Id - :param ip: - :param port: - :param disp: - :return: if Id is None, ServerItemsDetail object is returned. otherwise ServerItems list returned - """ - - if ServerId is not None: - TargetServer = ServerId - elif Name is not None: - TargetServer = Name - else: - TargetServer = None - - ItemsHeader = True - if TargetServer is not None: - Url = "/api/v1/Servers/%s" % TargetServer - ItemsHeader = False - ReturnType = ServerItemsDetailModule - else: - Url = "/api/v1/Servers" - ReturnType = ServerItemsModule - Params = dict() - Params['countPerPage'] = 100 - Conn = RestApi(Ip, Port, Url, authkey=ApiKey, params=Params, logger=logger) - Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) - if Res == ResOk: - AllServerInfo = list() - if Ret.Result == ResultSuccess: - if TargetServer is not None: - return Res, Errmsg, Ret, Ret.Data - else: - return Res, Errmsg, Ret, Ret.Data.Items - else: - return Res, Errmsg, Ret, AllServerInfo - else: - return Res, Errmsg, None, None - - -def GetAllServerDetailInfo(Ip, Port, Apikey, logger=None): - Res, Errmsg, Ret, Servers = GetServerInfo(Ip, Port, Apikey, logger=logger) - if Res == ResOk: - AllServerDetailInfo = list() - if Ret.Result == ResultSuccess: - for Svr in Servers: - Res, Errmsg, Ret, Detail = GetServerInfo(Ip, Port, Apikey, ServerId=Svr.Id, logger=logger) - if Res == ResOk: - AllServerDetailInfo.append(Detail) - else: - print(Errmsg) - return Res, Errmsg, Ret, AllServerDetailInfo - else: - print('fail to get Server List', Ret.Message) - return Res, Errmsg, Ret, None - else: - print('fail to get Server List', Errmsg) - return Res, Errmsg, Ret, None - - -@catch_exceptions() -def ShowServerInfo(Data, Id=None, Detail=False, SysinfoDisp=False): - if SysinfoDisp is True: - ServerTitleLine = '%s' % ('=' * 105) - ServerDataLine = '%s' % ('-' * 105) - title ="|%s|%s|%s|%s|" % ('Server(HostName)'.center(25), 'IpAddress'.center(15), 'Status'.center(17), ' ' * 43) - else: - if Detail: - ServerTitleLine = '%s' % ('=' * 148) - ServerDataLine = '%s' % ('-' * 148) - title ="|%s|%s|%s|%s|%s|%s|%s|" % ('HostName'.center(20), 'IpAddress'.center(15), 'Status'.center(10), - 'LoadAvg 1M 5M 15M'.center(15), 'MemTotal'.center(20), 'MemUsed'.center(20), 'Id'.center(38)) - else: - ServerTitleLine = '%s' % ('=' * 88) - ServerDataLine = '%s' % ('-' * 88) - title ="|%s|%s|%s|%s|" % ('HostName'.center(20), 'IpAddress'.center(15), 'Status'.center(10), 'Id'.center(38)) - - print(ServerTitleLine) - print(title) - print(ServerTitleLine) - if Id is None: - for svr in Data: - if len(svr.NetworkInterfaces) > 0: - ManagementIp = svr.NetworkInterfaces[0].IpAddress - else: - ManagementIp = '-' - - if SysinfoDisp is True: - _svr = "|%s|%s|%s|%s|" % (svr.Name.center(25), ManagementIp.center(15), str((svr.State)).center(17), ' ' * 43) - else: - if Detail: - _svr ="|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % \ - (svr.Name.center(20), ManagementIp.center(15), - DisplayState(svr.State).center(10), str(svr.LoadAverage1M).center(5), str(svr.LoadAverage5M).center(5) , - str(svr.LoadAverage15M).center(5), str(int(svr.MemoryTotal)).center(20), - str(int(svr.MemoryUsed)).center(20), svr.Id.center(38)) - else: - _svr = "|%s|%s|%s|%s|" % \ - (svr.Name.center(20), ManagementIp.center(15), DisplayState(svr.State).center(10), svr.Id.center(38)) - - print(_svr) - print(ServerDataLine) - - else: - svr = Data[0] - if Detail: - _svr = "|%s|%s|%s|%s|%s|%s|" % \ - (svr.Name.center(20), '{:30.30}'.format(svr.Description.center(30)), - '{:30.30}'.format(svr.CpuModel.center(30)), str(svr.Clock).center(20), - svr.State.center(20), svr.Id.center(30)) - else: - _svr = "|%s|%s|%s|" % (svr.Name.center(20), svr.State.center(20), svr.Id.center(30)) - print(_svr) - - -@catch_exceptions() -def AddServer(ip, port, ApiKey, Description, logger=None): - """ - register server info - :param ip: string - :param port: integer - :param Description: - :param logger: - :return:tuple(error code, error msg, Success to get result:header, fail to get result: None) - """ - server = RequestServerInfo(Description) - body = jsonpickle.encode(server, make_refs=False) - Url = '/api/v1/Servers' - ReturnType = ResponseHeaderModule - Params = body - Conn = RestApi(ip, port, Url, authkey=ApiKey, params=Params, logger=logger) - Res, Errmsg, Ret = Conn.post(ReturnType=ReturnType) - return Res, Errmsg, Ret - - - -@catch_exceptions() -def ServerInit(PortalIp, PortalPort, PortalApiKey, TargetServerIp, logger=None): - """ - register server info - :param ip: string - :param port: integer - :param Description: - :param logger: - :return:tuple(error code, error msg, Success to get result:header, fail to get result: None) - """ - """ - SshUser = get_input('Insert ssh user', str, default='root') - SshPassword = get_input('Insert ssh password', 'pwd', default='') - if PortalIp is None: # if ksanMonitor.conf is not configured - PortalIp = get_input('Insert portal host', str, default='') - PortalPort = get_input('Insert portal port', int, default='') - PortalApiKey = get_input('Insert portal api key', str, default='') - - if not IsIpValid(PortalIp): - Ret, Hostname, Ip = GetHostInfo(hostname=PortalIp) - if Ret is False: - print('Invalid Hostname') - sys.exit(-1) - else: - PortalIp = Ip - """ - SshUser='' - SshPassword = '' - server = RequestServerInitInfo() - server.Set(TargetServerIp, PortalIp, PortalPort, SshUser, SshPassword) - body = jsonpickle.encode(server, make_refs=False) - Url = '/api/v1/Servers/Initialize' - ReturnType = ResponseHeaderModule - Params = body - Conn = RestApi(PortalIp, PortalPort, Url, authkey=PortalApiKey, params=Params, logger=logger) - Res, Errmsg, Ret = Conn.post(ReturnType=ReturnType) - return Res, Errmsg, Ret - - -@catch_exceptions() -def RegisterServer(ip, port, ApiKey, Description, Name=None, logger=None): - """ - register server info to server pool - :param ip: string - :param port: integer - :param Description: - :param Name : In case of already exists, Search the server info with Name(hostname) - :param logger: - :return:if success or alredy registered, return tuple(error code, error msg, Header, ServerItems object), otherwise tuple(Res, Errmsg, None, None) - """ - server = RequestServerInfo(Description) - body = jsonpickle.encode(server, make_refs=False) - Url = '/api/v1/Servers' - ItemsHeader = False - ReturnType = ServerItemsDetailModule - Params = body - Conn = RestApi(ip, port, Url, authkey=ApiKey, params=Params, logger=logger) - - Res, Errmsg, Ret = Conn.post(ItemsHeader=ItemsHeader, ReturnType=ReturnType) - if Res == ResOk: - if Ret.Result == ResultSuccess: - return Res, Errmsg, Ret, Ret.Data - elif Ret.Code == CodeDuplicated: - res, errmsg, ret, AllServerInfo = GetServerInfo(ip, port, ApiKey, logger=logger) - if res == ResOk: - for svr in ret.Data.Items: - if svr.Name == Name: - return Res, Errmsg, Ret, svr - - return Res, Errmsg, Ret, None - - -@catch_exceptions() -def UpdateServerInfo(Ip, Port, Authkey, ServerId=None, Name=None, Description=None, State=None, logger=None): - """ - update server info - :param ip: - :param port: - :param Description: - :param Id: - :param logger: - """ - if ServerId is not None: - TargetServer = ServerId - elif Name is not None: - TargetServer = Name - else: - return ResInvalidCode, ResInvalidMsg, None - - Res, Errmsg, Ret, Data = GetServerInfo(Ip, Port, Authkey, ServerId=TargetServer, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - if Name is not None: - Data.Name = Name - if Description is not None: - Data.Description = Description - if State is not None: - Data.State = State - - Url = '/api/v1/Servers/%s' % TargetServer - server = UpdateServerInfoItems() - server.Set(Data.Name, Data.Description, Data.CpuModel, Data.Clock, Data.State, Data.Rack, Data.MemoryTotal) - body = jsonpickle.encode(server, make_refs=False) - Params = body - Conn = RestApi(Ip, Port, Url, authkey=Authkey, params=Params, logger=logger) - Ret, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResPonseHeader) - return Ret, Errmsg, Data - else: - return Res, Errmsg, Ret - else: - return Res, Errmsg, None - - -@catch_exceptions() -def UpdateServerUsage(Ip, Port, ServerId, LoadAverage1M=None, LoadAverage5M=None, LoadAverage15M=None, MemoryUsed=None, State=None, logger=None): - """ - update server info - :param ip: - :param port: - :param Description: - :param Id: - :param logger: - """ - Res, Errmsg, Ret, Data = GetServerInfo(Ip, Port, ServerId=ServerId, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - if LoadAverage1M is not None: - Data.LoadAverage1M = LoadAverage1M - if LoadAverage5M is not None: - Data.LoadAverage5M = LoadAverage5M - if LoadAverage15M is not None: - Data.LoadAverage15M = LoadAverage15M - if MemoryUsed is not None: - Data.MemoryUsed = MemoryUsed - - if State is not None: - Data.State = State - - Url = '/api/v1/Servers/%s' % ServerId - server = UpdateServerInfoItems() - server.Set(Data.Name, Data.Description, Data.CpuModel, Data.Clock, Data.State, Data.Rack, Data.MemoryTotal) - body = jsonpickle.encode(server, make_refs=False) - Params = body - Conn = RestApi(Ip, Port, Url, params=Params, logger=logger) - Ret, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResPonseHeader) - return Ret, Errmsg, Data - else: - return Res, Errmsg, Ret - else: - return Res, Errmsg, None - - -@catch_exceptions() -def UpdateServerState(Ip, Port, ServerId, State, logger=None): - """ - update server info - :param ip: - :param port: - :param Description: - :param Id: - :param logger: - """ - Url = '/api/v1/Servers/%s/State/%s' % (ServerId, State) - body = dict() - Params = body - Conn = RestApi(Ip, Port, Url, params=Params, logger=logger) - Ret, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResPonseHeader) - return Ret, Errmsg, Data - - -@catch_exceptions() -def RemoveServer(ip, port, ApiKey, ServerId=None, Name=None, logger=None): - """ - delete server info from server pool - :param ip: - :param port: - :param Id: - :param logger: - :return:tuple(error code, error msg, ResponseHeader class) - """ - - if ServerId is not None: - TargetServer = ServerId - elif Name is not None: - TargetServer = Name - else: - return ResInvalidCode, ResInvalidMsg + ' Id and Name are all None', None - - Url = '/api/v1/Servers/%s' % TargetServer - - ReturnType = ResponseHeaderModule - Conn = RestApi(ip, port, Url, authkey=ApiKey, logger=logger) - Res, Errmsg, Ret = Conn.delete(ReturnType=ReturnType) - return Res, Errmsg, Ret - - -@catch_exceptions() -def server_exist_check_with_servername(ip, port, Name, Authkey, ExcludeServerId=None, logger=None): - """ - check if a specific server exists with Name. - :param ip: - :param port: - :param Name: - :param ExcludeServerId: to exclude the server whose id is ExcludeServerId - :param logger: - :return:tuple(error code, error msg, Success to get result:True/False, fail to get result: None) - """ - server = RequestServerExistCheck(Name) - body = jsonpickle.encode(server, make_refs=False) - if ExcludeServerId is not None: - Url = '/api/v1/Servers/Exist/%s' % ExcludeServerId - else: - Url = '/api/v1/Servers/Exist' - Params = body - Conn = RestApi(ip, port, Url, authkey=Authkey, params=Params, logger=logger) - Res, Errmsg, Ret = Conn.post() - return Res, Errmsg, Ret - class RestHandlerServer: def __init__(self): self.logger = logging.getLogger(KsanAgentBinaryName) @@ -451,6 +66,7 @@ def InitAgentConf(self, Conf: AgentConf): f.flush() return SetRestReturn(ResultSuccess) + catch_exceptions() def AddServer(self, conf: AgentConf): """ @@ -482,179 +98,84 @@ def AddServer(self, conf: AgentConf): return SetRestReturn(Ret.Result, Code=Ret.Code, Message=Ret.Message) -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ifs_objstorage {init|add|delete|update|show|status} [option] - init : init management server ip and port - add : register local server to the management servers pool - delete -I : unregister local server from management servers pool - update -I : update server info - show -I : show the registered server info - check -n -I : check if the server exist. the server whose id is server id is excluded - [options] - -a : show all server info, default is true - -I : show specific server info with server id - -h, --help : show this help message and exit -""" - print(Usage) - -def ShowInfo(AllServers, AllDisks, ServerNetworks, Detail=None): - print("[ SERVER / NETWORK / DISK ]") - ServerTitleLine = "%s" % ('=' * 197) - print(ServerTitleLine) - ServerTitle = "|%s|%s|%s|%s|%s|%s|%s|%s|" % ('Name'.center(21), 'CpuModel'.center(31), - 'Clock'.center(21), 'State'.center(21), 'LoadAvg(1M 5M 15M)'.center(19), - 'MemTotal'.center(21), 'MemUsed'.center(21), 'Id'.center(31)) - print(ServerTitle) - print(ServerTitleLine) - - ServerDataLine = "%s" % ('-' * 197) - - for idx, Svr in enumerate(AllServers): - if idx > 0: - print(ServerDataLine) - ServerData = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % \ - (Svr.Name.center(21), - '{:31.31}'.format(Svr.CpuModel.center(31)), str(Svr.Clock).center(21), - Svr.State.center(21), str(Svr.LoadAverage1M).center(5), str(Svr.LoadAverage5M).center(5), - str(Svr.LoadAverage15M).center(5), str(int(Svr.MemoryTotal)).center(20), - str(int(Svr.MemoryUsed)).center(20), Svr.Id.center(30)) - print(ServerData) - print(ServerDataLine) - - NetworkTitle = "%s|%s|%s|%s|%s|%s|%s|" % (' '.center(70), 'Network Device'.center(20) , - 'IpAddress'.center(20), 'Rx'.center(10), 'Tx'.center(10), - 'LinkStatus'.center(20), 'Id'.center(40)) - print(NetworkTitle) - NetworkTitleLine = "%s%s" % (' ' * 70, '-' * 127) - print(NetworkTitleLine) - - for interface in ServerNetworks[Svr.Id]: - NetData = "%s|%s|%s|%s|%s|%s|%s|" % (' '.center(70), - '{:20.20}'.format(interface.Name.center(20)), - '{:20.20}'.format(interface.IpAddress.center(20)), - str(int(interface.Rx)).center(10), - str(int(interface.Tx)).center(10), interface.LinkState.center(20), - interface.Id.center(40),) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(NetData) - - DiskTitleLine = "%s%s" % (' ' * 54, '-' * 143) - print(DiskTitleLine) - DiskTitle = "%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 54, 'DiskId'.center(40), 'Path'.center(15), - 'State'.center(10), 'Total'.center(20), 'Used'.center(20), 'Free'.center(20), 'RwMode'.center(10)) - print(DiskTitle) - print(DiskTitleLine) - for disk in AllDisks: - DiskSvr = DeserializeResponse(disk.Server, ServerItemsModule) - if Svr.Id != DiskSvr.Id: - continue - - # svr = GetDataFromBody(disk.Server, ServerItemsModule) - DiskData = "%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 54, str(disk.Id).center(40), - '{:15.15}'.format(disk.Path.center(15)), - disk.State.center(10), str(int(disk.TotalSize)).center(20), - str(int(disk.UsedSize)).center(20), - str(int(disk.TotalSize - disk.UsedSize - disk.ReservedSize)).center(20), - disk.RwMode.center(10)) - print(DiskData) - - print(ServerTitleLine) - print() - """ - print("[ NETWORK ]") - compart = "%s" % ('-' * 209) - print(compart) - title ="|%s|%s|%s|%s|%s|%s|%s|%s|" % ('Id'.center(40), 'Server Name'.center(40), 'Network Device'.center(20), - 'IpAddress'.center(20), 'Rx'.center(10), 'Tx'.center(10), - 'LinkStatus'.center(20), 'ServerId'.center(40)) - - print(title) - print(compart) - for Svr in AllServers: - for interface in ServerNetworks[Svr.Id]: - _nic =" %s %s %s %s %s %s %s %s" % (interface.Id.center(40), '{:40.40}'.format(Svr.Name.center(40)), - '{:20.20}'.format(interface.Name.center(20)), - '{:20.20}'.format(interface.IpAddress.center(20)), - str(int(interface.Rx)).center(10), str(int(interface.Tx)).center(10), interface.LinkState.center(20), Svr.Id.center(40)) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(_nic) - """ - -#main() - - -def ShowSysInfo(AllServersDetail): - print("[ SERVER / NETWORK / DISK ]") - ServerTitleLine = "%s" % ('=' * 197) - print(ServerTitleLine) - ServerTitle = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ('Name'.center(21), 'CpuModel'.center(20), - 'Clock'.center(10), 'State'.center(10), - 'LoadAvg(1M 5M 15M)'.center(19), - 'MemTotal(MB)'.center(15), 'MemUsed(MB)'.center(15), 'Id'.center(38), - ' '.center(39)) - print(ServerTitle) - print(ServerTitleLine) - - ServerDataLine = "%s" % ('-' * 197) - - for idx, Svr in enumerate(AllServersDetail): - if idx > 0: - print(ServerDataLine) - ServerData = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % \ - (Svr.Name.center(21), - '{:20.20}'.format(Svr.CpuModel.center(20)), str(Svr.Clock).center(10), - Svr.State.center(10), str(Svr.LoadAverage1M).center(6), str(Svr.LoadAverage5M).center(5), - str(Svr.LoadAverage15M).center(6), str(int(Svr.MemoryTotal/1000000)).center(15), - str(int(Svr.MemoryUsed/1000000)).center(15), Svr.Id.center(38)) - print(ServerData) - print(ServerDataLine) - - DiskTitleLine = "%s%s" % (' ' * 21, '-' * 176) - DiskTitle = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 21, 'DiskId'.center(40), 'Path'.center(15), - 'State'.center(10), 'Total Size(MB)'.center(15), - 'Used Size(MB)'.center(15), - 'Free Size(MB)'.center(15), 'Total Inode'.center(15), - 'Used Inode'.center(15), 'Free Inode'.center(15), - 'RwMode'.center(10)) - print(DiskTitle) - print(DiskTitleLine) - - for disk in Svr.Disks: - # svr = GetDataFromBody(disk.Server, ServerItemsModule) - DiskData = "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 21, str(disk.Id).center(40), - '{:15.15}'.format(disk.Path.center(15)), - disk.State.center(10), str(int(disk.TotalSize/1000000)).center(15), - str(int(disk.UsedSize/1000000)).center(15), - str(int( - (disk.TotalSize - disk.UsedSize - disk.ReservedSize)/1000000)).center( - 15), str(int(disk.TotalInode)).center(15), str(int(disk.UsedInode)).center(15), str(int( - disk.TotalInode - disk.UsedInode - disk.ReservedInode)).center( - 15), - disk.RwMode.center(10)) - print(DiskData) - print(DiskTitleLine) - - - - NetworkTitle = "%s|%s|%s|%s|%s|%s|%s|" % (' '.center(70), 'Network Device'.center(20) , - 'IpAddress'.center(20), 'Rx'.center(10), 'Tx'.center(10), - 'LinkStatus'.center(20), 'Id'.center(40)) - print(NetworkTitle) - NetworkTitleLine = "%s%s" % (' ' * 70, '-' * 127) - print(NetworkTitleLine) - - for net in Svr.NetworkInterfaces: - NetData = "%s|%s|%s|%s|%s|%s|%s|" % (' '.center(70), - '{:20.20}'.format(net.Name.center(20)), - '{:20.20}'.format(net.IpAddress.center(20)), - str(int(net.Rx)).center(10), - str(int(net.Tx)).center(10), net.LinkState.center(20), - net.Id.center(40),) # svr.ModDate.center(20), svr.ModId.center(20), svr.ModName.center(20), svr.Id.center(30)) - print(NetData) - - - print(ServerTitleLine) - print() +def UpdateServerInfoDetail(PortalIp, PortalPort, PortalApiKey, ServerId, TargetNetwork=None, logger=None): + + # get server info + Res, Errmsg, Ret, ServerData = GetServerInfo(PortalIp, PortalPort, PortalApiKey, ServerId=ServerId, logger=logger) + if Res == ResOk: + if Ret.Result == ResultSuccess: + + if ServerData is not None: + InputServerInfo = [{'key': 'Name', 'value': ServerData.Name, 'type': str, 'question': 'Insert new server name'}, + {'key': 'Description', 'value': ServerData.Description, 'type': str, + 'question': 'Insert new server\'s description'} + ] + for info in InputServerInfo: + + QuestionString = info['question'] + ValueType = info['type'] + DefaultValue = info['value'] + ServerData.__dict__[info['key']] = get_input(QuestionString, ValueType, DefaultValue) + + Url = '/api/v1/Servers/%s' % ServerId + server = UpdateServerInfoItems() + server.Set(ServerData.Name, ServerData.Description, ServerData.CpuModel, ServerData.Clock, ServerData.State, ServerData.Rack, + ServerData.MemoryTotal) + body = jsonpickle.encode(server, make_refs=False) + Params = body + Conn = RestApi(PortalIp, PortalPort, Url, authkey=PortalApiKey, params=Params, logger=logger) + Ret, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResPonseHeader) + if Ret != ResOk: + return Ret, Errmsg + else: + if Data.Result != ResultSuccess: + return Data.Result, Data.Message + else: + return Ret.Result, Ret.Message + else: + return Res, Errmsg + + # get network info + if TargetNetwork is not None: + Res, Errmsg, NetworkData, ServerId = GetNetworkInterfaceDetailWithId(PortalIp, PortalPort, PortalApiKey, TargetNetwork) + if Res == ResOk: + InputNetworkInfo = [{'key': 'Name', 'value': NetworkData.Name, 'type': str, 'question': 'Insert new network name'}, + {'key': 'Description', 'value': NetworkData.Description, 'type': str, + 'question': 'Insert new network\'s description'}, + {'key': 'MacAddress', 'value': NetworkData.MacAddress, 'type': str, + 'question': 'Insert new network\'s mac address'}, + {'key': 'IpAddress', 'value': NetworkData.IpAddress, 'type': 'ip', + 'question': 'Insert new network\'s ipaddress'}, + {'key': 'SubnetMask', 'value': NetworkData.SubnetMask, 'type': str, + 'question': 'Insert new network\'s subnet mask'}, + {'key': 'Gateway', 'value': NetworkData.Gateway, 'type': 'ip', + 'question': 'Insert new network\'s gateway'}, + {'key': 'Dns1', 'value': NetworkData.Dns1, 'type': str, 'question': 'Insert new network\'s dns1'}, + {'key': 'Dns2', 'value': NetworkData.Dns2, 'type': str, 'question': 'Insert new network\'s dns2'}, + {'key': 'Bandwidth', 'value': NetworkData.BandWidth, 'type': int, + 'question': 'Insert new network\'s bandwidth'}, + ] + for info in InputNetworkInfo: + QuestionString = info['question'] + ValueType = info['type'] + DefaultValue = info['value'] + NetworkData.__dict__[info['key']] = get_input(QuestionString, ValueType, DefaultValue) + + body = jsonpickle.encode(NetworkData, make_refs=False) + Url = '/api/v1/Servers/%s/NetworkInterfaces/%s' % (ServerId, TargetNetwork) + Params = body + Conn = RestApi(PortalIp, PortalPort, Url, authkey=PortalApiKey, params=Params, logger=logger) + Ret, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResPonseHeader) + if Ret != ResOk: + return Ret, Errmsg + else: + return Data.Result, Data.Message + else: + return Res, Errmsg + else: + return Res, ResultSuccess + + def ServerUtilHandler(Conf, Action, Parser, logger): @@ -714,12 +235,9 @@ def ServerUtilHandler(Conf, Action, Parser, logger): Parser.print_help() sys.exit(-1) - Res, Errmsg, Ret = UpdateServerInfo(PortalIp, PortalPort, PortalApiKey, Name=options.ServerName, - Description=options.Description, State=options.State, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) + Res, Errmsg = UpdateServerInfoDetail(PortalIp, PortalPort, PortalApiKey, options.ServerName, + TargetNetwork=options.NetworkName, logger=logger) + print(Errmsg) elif Action.lower() == 'update_state': if options.ServerName is None: @@ -738,7 +256,14 @@ def ServerUtilHandler(Conf, Action, Parser, logger): Res, errmsg, Ret, Data = GetAllServerDetailInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) if Res == ResOk: if Ret.Result == ResultSuccess: - ShowServerInfo(Data, Detail=options.Detail) + if options.MoreDetail: + Detail = MoreDetailInfo + elif options.Detail: + Detail = DetailInfo + else: + Detail = SimpleInfo + + ShowServerInfo(Data, Detail=Detail) else: print(Ret.Result, Ret.Message) else: diff --git a/core/common/server/server_mq_handle.py b/core/common/server/server_mq_handle.py index 5c227c1f..8414b875 100644 --- a/core/common/server/server_mq_handle.py +++ b/core/common/server/server_mq_handle.py @@ -11,14 +11,15 @@ """ import os, sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) #from common.common.mqutils import Mq -from server.server_manage import * +from server.server_api import GetServerUsage +import mqmanage.mq_api import mqmanage.mq from const.common import * from const.server import ServerUsageItems, ServerStateItems -from common.init import GetConf, WaitAgentConfComplete, GetAgentConfig -from common.shcommand import UpdateEtcHosts +from common.base_utils import GetConf, WaitAgentConfComplete, GetAgentConfig from const.mq import MqVirtualHost, RoutKeyServerUsage, ExchangeName, RoutKeyServerState, \ RoutKeyServerUpdateFinder, RoutKeyServerAddFinder, RoutKeyServerDelFinder import socket @@ -30,8 +31,8 @@ def MonUpdateServerUsage(Conf, logger): Conf = WaitAgentConfComplete(inspect.stack()[1][3], logger) conf = GetAgentConfig(Conf) - ServerUsageMq = mqmanage.mq.Mq(conf.MQHost, int(conf.MQPort), MqVirtualHost, conf.MQUser, conf.MQPassword, RoutKeyServerUsage, ExchangeName) - ServerStateMq = mqmanage.mq.Mq(conf.MQHost, int(conf.MQPort), MqVirtualHost, conf.MQUser, conf.MQPassword, RoutKeyServerState, ExchangeName) + ServerUsageMq = mqmanage.mq.Mq(conf.MQHost, int(conf.MQPort), MqVirtualHost, conf.MQUser, conf.MQPassword, RoutKeyServerUsage, ExchangeName, logger=logger) + ServerStateMq = mqmanage.mq.Mq(conf.MQHost, int(conf.MQPort), MqVirtualHost, conf.MQUser, conf.MQPassword, RoutKeyServerState, ExchangeName, logger=logger) ServerMonitorInterval = int(conf.ServerMonitorInterval)/1000 while True: svr = GetServerUsage(conf.ServerId) @@ -50,55 +51,3 @@ def MonUpdateServerUsage(Conf, logger): ServerStateMq.Sender(Mqsend) time.sleep(ServerMonitorInterval) - - -def MqServerHandler(Conf, RoutingKey, Body, Response, ServerId, logger): - logger.debug("MqServerHandler %s %s" % (str(RoutingKey), str(Body))) - try: - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Body = Body.decode('utf-8') - Body = json.loads(Body) - body = DictToObject(Body) - if RoutKeyServerUpdateFinder.search(RoutingKey) or RoutKeyServerAddFinder.search(RoutingKey): - #IpAddress = body.NetworkInterfaces.IpAddress # not available - pass - - elif RoutKeyServerDelFinder.search(RoutingKey): - HostName = body.Name - HostInfo = [('', HostName)] - UpdateEtcHosts(HostInfo, 'remove') - logging.log(logging.INFO, 'host is removed. %s' % str(HostInfo)) - if ServerId == body.Id: - ret, errlog = RemoveQueue(Conf) - if ret is False: - logging.error('fail to remove queue %s' % errlog) - else: - logging.log(logging.INFO, 'success to remove queue') - if os.path.exists(MonServicedConfPath): - os.unlink(MonServicedConfPath) - logging.log(logging.INFO, 'ksanMonitor.conf is removed') - - ''' - if RoutKeyServerUpdateFinder.search(RoutingKey): - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Body = Body.decode('utf-8') - Body = json.loads(Body) - body = DictToObject(Body) - if ServerId == body.ServerId: - ret = CheckDiskMount(body.Path) - if ret is False: - ResponseReturn = mqmanage.mq.MqReturn(ret, Code=1, Messages='No such disk is found') - Response.IsProcessed = True - print(ResponseReturn) - return ResponseReturn - ''' - return ResponseReturn - except Exception as err: - print(err) - -def RemoveQueue(Conf): - QueueHost = Conf.MQHost - QueueName = 'ksan-agent-%s' % Conf.ServerId - mqmanage.mq.RemoveQueue(QueueHost, QueueName) - return True, '' - diff --git a/core/common/service/control.py b/core/common/service/control.py index fe5eee97..c134b806 100644 --- a/core/common/service/control.py +++ b/core/common/service/control.py @@ -12,13 +12,9 @@ import os, sys import time -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from const.common import * -from common.utils import IsDaemonRunning, CheckParams -from common.shcommand import * -from service.service_manage import AddService -import xml.etree.ElementTree as ET -import signal +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from common.utils import * class ServiceUnit: def __init__(self, logger, ServiceType): @@ -34,6 +30,10 @@ def GetServiceUnit(self): self.ServiceUnit = SystemdKsanOSDServiceName elif self.ServiceType.lower() == TypeServiceLifecycle.lower(): self.ServiceUnit = SystemdKsanLifecycleServiceName + elif self.ServiceType.lower() == TypeServiceReplication.lower(): + self.ServiceUnit = SystemdKsanReplicationServiceName + elif self.ServiceType.lower() == TypeServiceLogManager.lower(): + self.ServiceUnit = SystemdKsanLogManagerServiceName def Start(self): Cmd = 'systemctl start %s' % self.ServiceUnit diff --git a/core/common/service/gw.py b/core/common/service/gw.py deleted file mode 100644 index cc86d16d..00000000 --- a/core/common/service/gw.py +++ /dev/null @@ -1,119 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os, sys -import time -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import * -from common.utils import IsDaemonRunning, CheckParams -from common.shcommand import * -from service.service_manage import AddService -import xml.etree.ElementTree as ET -import signal - - -class KsanGW: - def __init__(self, logger): - self.logger = logger - self.MonConf = None - self.GwConf = None - self.ObjmanagerConf = None - self.apache_pid_path = '/var/run/ksanGw.pid' - self.s3gw_pid_path = '/var/run/ksanGw.pid' - - - def PreRequest(self): - # java version chck: - out, err = shcall("java -version") - out += str(err) - if 'openjdk version "11.' not in out: - return False, 'java11 is not installed' - KsanGwBinPath = "%s/%s" % (KsanBinPath, KsanGwBinaryName) - if not os.path.exists(KsanGwBinPath): - return False, '%s is not found' % KsanGwBinPath - - return True, '' - - def Start(self): - ret, errmsg = self.PreRequest() - if ret is False: - return ret, errmsg - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - print('already Running') - return False, 'ksanGw is already running' - else: - StartCmd = 'cd %s; nohup java -jar -Dlogback.configurationFile=%s %s >/dev/null 2>&1 &' % \ - (KsanBinPath, GwXmlFilePath, KsanGwBinaryName) - self.logger.debug(StartCmd) - os.system(StartCmd) - time.sleep(2) - - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - return True, '' - else: - return False, 'Fail to start ksanGw' - - def Stop(self, Force=False): - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is False: - if Force is True: - return True, '' - else: - return False, 'ksanGw is not running' - else: - - try: - self.logger.debug('kill %s' % KsanGwBinaryName) - os.kill(int(Pid), signal.SIGTERM) - os.unlink(KsanGwPidFile) - print('Done') - return True, '' - except OSError as err: - return False, 'Fail to stop. %s' % err - - def Restart(self): - RetryCnt = 0 - while True: - RetryCnt += 1 - Ret, ErrMsg = self.Stop(Force=True) - if Ret is False: - if RetryCnt > ServiceContolRetryCount: - return False, ErrMsg - else: - time.sleep(IntervalShort) - else: - break - - RetryCnt = 0 - while True: - RetryCnt += 1 - Ret, ErrMsg = self.Start() - if Ret is False: - if RetryCnt > ServiceContolRetryCount: - return False, ErrMsg - else: - time.sleep(IntervalShort) - else: - return Ret, ErrMsg - - - def Status(self): - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - print('KsanGw ... Ok') - return True, 'KsanGw ... Ok' - else: - print('KsanGw ... Not Ok') - return False, 'KsanGw ... Not Ok' - diff --git a/core/common/service/lifecycle.py b/core/common/service/lifecycle.py deleted file mode 100644 index 9f853781..00000000 --- a/core/common/service/lifecycle.py +++ /dev/null @@ -1,100 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os, sys -import time -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import * -from common.utils import IsDaemonRunning, CheckParams -from common.shcommand import * -from service.service_manage import AddService -import xml.etree.ElementTree as ET -import signal - - - - -class KsanLifecycle: - def __init__(self, logger): - self.logger = logger - self.MonConf = None - self.GwConf = None - self.ObjmanagerConf = None - self.apache_pid_path = '/var/run/ksanGw.pid' - self.s3gw_pid_path = '/var/run/ksanGw.pid' - - - def PreRequest(self): - # java version chck: - out, err = shcall("java -version") - out += str(err) - if 'openjdk version "11.' not in out: - return False, 'java11 is not installed' - KsanGwBinPath = "%s/%s" % (KsanBinPath, KsanGwBinaryName) - if not os.path.exists(KsanGwBinPath): - return False, '%s is not found' % KsanGwBinPath - - return True, '' - - def Start(self): - ret, errmsg = self.PreRequest() - if ret is False: - return ret, errmsg - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - print('already running') - return False, 'ksanGw is already running' - else: - StartCmd = 'cd %s; nohup java -jar -Dlogback.configurationFile=%s %s >/dev/null 2>&1 &' % \ - (KsanBinPath, GwXmlFilePath, KsanGwBinaryName) - os.system(StartCmd) - time.sleep(2) - - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - return True, '' - else: - return False, 'Fail to start ksanGw' - - def Stop(self): - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is False: - return False, 'ksanGw is not running' - else: - - try: - os.kill(int(Pid), signal.SIGTERM) - os.unlink(KsanGwPidFile) - print('Done') - return True, '' - except OSError as err: - return False, 'Fail to stop. %s' % err - - def Restart(self): - Ret, ErrMsg = self.Stop() - if Ret is not True: - print(ErrMsg) - Ret, ErrMsg = self.Start() - if Ret is not True: - print(ErrMsg) - else: - print('Done') - - - def Status(self): - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - print('KsanGw ... Ok') - return True, 'KsanGw ... Ok' - else: - print('KsanGw ... Not Ok') - return False, 'KsanGw ... Not Ok' diff --git a/core/common/service/metering.py b/core/common/service/metering.py deleted file mode 100644 index c88ae6e2..00000000 --- a/core/common/service/metering.py +++ /dev/null @@ -1,101 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os, sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import * -from common.utils import IsDaemonRunning, CheckParams -from common.shcommand import * -import time -from service.service_manage import AddService -import xml.etree.ElementTree as ET -import signal - - - - - -class KsanMetering: - def __init__(self, logger): - self.logger = logger - self.MonConf = None - self.GwConf = None - self.ObjmanagerConf = None - self.apache_pid_path = '' - self.recovery_pid_path = '' - - - def PreRequest(self): - # java version chck: - out, err = shcall("java -version") - out += str(err) - if 'openjdk version "11.' not in out: - return False, 'java11 is not installed' - KsanGwBinPath = "%s/%s" % (KsanBinPath, KsanGwBinaryName) - if not os.path.exists(KsanGwBinPath): - return False, '%s is not found' % KsanGwBinPath - - return True, '' - - def Start(self): - ret, errmsg = self.PreRequest() - if ret is False: - return ret, errmsg - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - print('already running') - return False, 'ksanGw is already running' - else: - StartCmd = 'cd %s; nohup java -jar -Dlogback.configurationFile=%s %s >/dev/null 2>&1 &' % \ - (KsanBinPath, GwXmlFilePath, KsanGwBinaryName) - os.system(StartCmd) - time.sleep(2) - - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - return True, '' - else: - return False, 'Fail to start ksanGw' - - def Stop(self): - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is False: - return False, 'ksanGw is not running' - else: - - try: - os.kill(int(Pid), signal.SIGTERM) - os.unlink(KsanGwPidFile) - print('Done') - return True, '' - except OSError as err: - return False, 'Fail to stop. %s' % err - - def Restart(self): - Ret, ErrMsg = self.Stop() - if Ret is not True: - print(ErrMsg) - Ret, ErrMsg = self.Start() - if Ret is not True: - print(ErrMsg) - else: - print('Done') - - - def Status(self): - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - print('KsanGw ... Ok') - return True, 'KsanGw ... Ok' - else: - print('KsanGw ... Not Ok') - return False, 'KsanGw ... Not Ok' diff --git a/core/common/service/mongo.py b/core/common/service/mongo.py deleted file mode 100644 index 8457fa2e..00000000 --- a/core/common/service/mongo.py +++ /dev/null @@ -1,157 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -from common.init import * -from common.utils import IsDaemonRunning -import time -import json -from const.common import DictToObject -from common.init import GetConf, GetHostInfo -from common.shcommand import * -from service.service_manage import GetServiceMongoDBConfig -import logging -import xml.etree.ElementTree as ET -import signal - - -class KsanMongoDB: - def __init__(self, logger): - self.logger = logger - self.ObjmanagerConf = None - self.apache_pid_path = '' - self.s3gw_pid_path = '' - self.MonConf = None - self.DbConf = None - self.isLocalPrimary = False - self.GetMonConf() - - def GetMonConf(self): - if not os.path.exists(MonServicedConfPath): - return False, '%s is not found' % MonServicedConfPath - - ret, self.MonConf = GetConf(MonServicedConfPath) - if ret is False: - return False, 'fail to get %s' % MonServicedConfPath - else: - return True, '' - - def GetMongoDBConf(self): - Retry = 0 - while True: - Retry += 1 - Res, ErrMsg, Ret, Data = GetServiceMongoDBConfig(self.MonConf.PortalHost, int(self.MonConf.PortalPort), - self.MonConf.PortalApiKey, logger=self.logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - DbConf = json.loads(Data.Config) - self.DbConf = DictToObject(DbConf) - self.isLocalPrimayNode() - return True, '' - else: - if Retry > 2: - return False, 'fail to get MongoDB config %s' % ErrMsg - else: - time.slee(1) - else: - if Retry > 2: - return False, 'fail to get MongoDB config' - else: - time.sleep(1) - - def isLocalPrimayNode(self): - Ret, Hostname, Ip = GetHostInfo() - if self.DbConf.PrimaryHostName == Hostname: - self.isLocalPrimary = True - - def PreRequest(self): - # java version chck: - if not os.path.exists(KsanMongDbManagerBinPath): - return False, '%s is not installed' % KsanMongDbManagerBinPath - - MongoScriptDir = '/usr/local/ksan/mongo' - isValid = True - for scrip in ['configd.start', 'configd.stop', 'mongos.start', 'mongos.stop', 'shard.start', 'shard.stop']: - full_path = '%s/%s' % (MongoScriptDir, scrip) - if not os.path.exists(full_path): - isValid = False - - if isValid is False: - Shard1Port = str(self.DbConf.Shard1Port) - Shard2Port = str(self.DbConf.Shard2Port) - ConfigServerPort = str(self.DbConf.ConfigServerPort) - MongoDbPort = str(self.DbConf.MongoDbPort) - HomeDir = self.DbConf.HomeDir - - Option = " -P %s -S %s -C %s -D %s -H %s %s " % (Shard1Port, Shard2Port, ConfigServerPort, MongoDbPort, - HomeDir, '-R' if self.isLocalPrimary is True else '' ) - InitCmd = '%s init %s' % (KsanMongDbManagerBinPath, Option) - out, err = shcall(InitCmd) - logging.log(logging.INFO, "%s %s %s" % (InitCmd, str(out), str(err))) - - isValid = True - - if not os.path.exists('/usr/bin/mongos'): - isValid = False - - if isValid is False: - return False, 'mongodb is not initialized' - - return True, '' - - def Start(self): - #ret, errmsg = self.PreRequest() - #if ret is False: - # return ret, errmsg - MongoDbStartCmd = 'nohup %s start &' % KsanMongDbManagerBinPath - os.system(MongoDbStartCmd) - return True, 'MongoDB ... Ok' - - - def Stop(self): - MongoDbStopCmd = '%s stop '% KsanMongDbManagerBinPath - out, err = shcall(MongoDbStopCmd) - RetryCnt = 0 - while True: - RetryCnt += 1 - if 'Ok' not in out: - return True, 'MongoDB ... Ok' - else: - if RetryCnt > 5: - return False, 'MongoDB ... Not Ok' - else: - time.sleep(2) - - def Restart(self): - MongoDbRestartCmd = '%s stop; %s start' % (KsanMongDbManagerBinPath, KsanMongDbManagerBinPath) - out, err = shcall(MongoDbRestartCmd) - RetryCnt = 0 - while True: - RetryCnt += 1 - if 'Ok' in out: - return True, 'MongoDB ... Ok' - else: - if RetryCnt > 5: - return False, 'MongoDB ... Not Ok' - else: - time.sleep(2) - - def Status(self): - #Ret, Pid = IsDaemonRunning(KsanGwPidPath, CmdLine=KsanGwBinaryName) - MongoDbStatusCmd = '%s status' % KsanMongDbManagerBinPath - out, err = shcall(MongoDbStatusCmd) - if 'Ok' in out: - print('MongoDB ... Ok') - return True, 'MongoDB ... Ok' - else: - print('MongoDB ... Not Ok') - return False, 'MongoDB ... Not Ok' - diff --git a/core/common/service/monitor.py b/core/common/service/monitor.py deleted file mode 100644 index f9928c88..00000000 --- a/core/common/service/monitor.py +++ /dev/null @@ -1,88 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os, sys -import time -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import * -from common.utils import IsDaemonRunning -from common.shcommand import * -import signal - - -class KsanMonitor: - def __init__(self, logger): - self.logger = logger - self.MonConf = None - self.OsdConf = None - self.BinaryName = KsanMonitorBinaryName - self.PidFile = KsanMonitorPidFile - - def PreRequest(self): - # java version chck: - out, err = shcall("java -version") - out += str(err) - if 'openjdk version "11.' not in out: - return False, 'java11 is not installed' - KsanOsdBinPath = "%s/%s" % (KsanBinPath, KsanOsdBinaryName) - if not os.path.exists(KsanOsdBinPath): - return False, '%s is not found' % KsanOsdBinPath - - return True, '' - - - def Start(self): - ret, errmsg = self.PreRequest() - if ret is False: - return ret, errmsg - - Ret, Pid = IsDaemonRunning(self.PidFile, CmdLine=self.BinaryName) - if Ret is True: - print('Already Running') - return False, '%s is already running' % self.BinaryName - else: - StartCmd = '%s/%s start' % \ - (KsanBinPath, self.BinaryName) - os.system(StartCmd) - time.sleep(2) - - Ret, Pid = IsDaemonRunning(self.PidFile, CmdLine=self.BinaryName) - if Ret is True: - return True, '' - else: - return False, 'Fail to start %s' % self.BinaryName - - def Stop(self): - Ret, Pid = IsDaemonRunning(self.PidFile, CmdLine=self.BinaryName) - if Ret is False: - return False, '%s is not running' % self.BinaryName - else: - try: - os.kill(int(Pid), signal.SIGTERM) - os.unlink(self.PidFile) - print('Done') - return True, '' - except OSError as err: - return False, 'Fail to stop. %s' % err - - def Restart(self): - self.Stop() - self.Start() - - def Status(self): - Ret, Pid = IsDaemonRunning(self.PidFile, CmdLine=self.BinaryName) - if Ret is True: - print('%s ... Ok' % self.BinaryName) - return True, '%s ... Ok' % self.BinaryName - else: - print('%s ... Not Ok' % self.BinaryName) - return False, '%s ... Not Ok' % self.BinaryName diff --git a/core/common/service/osd.py b/core/common/service/osd.py deleted file mode 100644 index 902c2bce..00000000 --- a/core/common/service/osd.py +++ /dev/null @@ -1,144 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import os, sys -import time -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import * -from common.utils import IsDaemonRunning -from common.shcommand import * -from service.service_manage import AddService -import signal - - - -class KsanOsdConfig: - def __init__(self, LocalIp): - self.pool_size = 650 - self.port = 8000 - self.ec_schedule_minutes = 30000 - self.ec_apply_minutes = 3000000 - self.ec_file_size = 100000 - self.cache_disk = '' - self.cache_schedule_minutes = 2 - self.cache_file_size = 1024 - self.cache_limit_minutes = 5 - self.trash_schedule_minutes = 5 - - def Set(self, pool_size, port, ec_schedule_minutes, ec_apply_minutes, ec_file_size, cache_disk, - cache_schedule_minutes, cache_file_size, cache_limit_minutes, trash_schedule_minutes): - - self.pool_size = pool_size - self.port = port - self.ec_schedule_minutes = ec_schedule_minutes - self.ec_apply_minutes = ec_apply_minutes - self.ec_file_size = ec_file_size - self.cache_disk = cache_disk - self.cache_schedule_minutes = cache_schedule_minutes - self.cache_file_size = cache_file_size - self.cache_limit_minutes = cache_limit_minutes - self.trash_schedule_minutes = trash_schedule_minutes - - -class KsanOsd: - def __init__(self, logger): - self.logger = logger - self.MonConf = None - self.OsdConf = None - - def PreRequest(self): - # java version chck: - out, err = shcall("java -version") - out += str(err) - if 'openjdk version "11.' not in out: - return False, 'java11 is not installed' - KsanOsdBinPath = "%s/%s" % (KsanBinPath, KsanOsdBinaryName) - if not os.path.exists(KsanOsdBinPath): - return False, '%s is not found' % KsanOsdBinPath - - return True, '' - - - def Start(self): - ret, errmsg = self.PreRequest() - if ret is False: - return ret, errmsg - - Ret, Pid = IsDaemonRunning(KsanOsdPidFile, CmdLine=KsanOsdBinaryName) - if Ret is True: - print('already running') - return False, 'ksanOsd is already running' - else: - StartCmd = 'cd %s; nohup java -jar -Dlogback.configurationFile=%s %s >/dev/null 2>&1 &' % \ - (KsanBinPath, OsdXmlFilePath, KsanOsdBinaryName) - self.logger.debug(StartCmd) - os.system(StartCmd) - time.sleep(2) - - Ret, Pid = IsDaemonRunning(KsanOsdPidFile, CmdLine=KsanOsdBinaryName) - if Ret is True: - return True, '' - else: - return False, 'Fail to start ksanOsd' - - def Stop(self, Force=False): - Ret, Pid = IsDaemonRunning(KsanOsdPidFile, CmdLine=KsanOsdBinaryName) - if Ret is False: - if Force is True: - return True, '' - else: - return False, 'ksanOsd is not running' - else: - try: - self.logger.debug('kill %s' % KsanOsdBinaryName) - os.kill(int(Pid), signal.SIGTERM) - os.unlink(KsanOsdPidFile) - print('Done') - return True, '' - except OSError as err: - return False, 'Fail to stop. %s' % err - - def Restart(self): - RetryCnt = 0 - while True: - RetryCnt += 1 - Ret, ErrMsg = self.Stop(Force=True) - if Ret is False: - if RetryCnt > ServiceContolRetryCount: - return False, ErrMsg - else: - time.sleep(IntervalShort) - else: - break - - RetryCnt = 0 - while True: - RetryCnt += 1 - Ret, ErrMsg = self.Start() - if Ret is False: - if RetryCnt > ServiceContolRetryCount: - return False, ErrMsg - else: - time.sleep(IntervalShort) - else: - return Ret, ErrMsg - - - def Status(self): - Ret, Pid = IsDaemonRunning(KsanOsdPidFile, CmdLine=KsanOsdBinaryName) - if Ret is True: - print('ksanOsd ... Ok') - return True, 'KsanOsd ... Ok' - else: - print('ksanOsd ... Not Ok') - return False, 'KsanOsd ... Not Ok' diff --git a/core/common/service/recovery.py b/core/common/service/recovery.py deleted file mode 100644 index c7273d4a..00000000 --- a/core/common/service/recovery.py +++ /dev/null @@ -1,95 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os, sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -import time -from common.init import * -from common.utils import IsDaemonRunning, IsDaemonRunningWithSystemd -from common.shcommand import * -from service.service_manage import AddService -import xml.etree.ElementTree as ET -import signal - - -class KsanRecovery: - def __init__(self, logger): - self.logger = logger - self.MonConf = None - self.GwConf = None - self.ObjmanagerConf = None - - def PreRequest(self): - # java version chck: - out, err = shcall("java -version") - out += str(err) - if 'openjdk version "11.' not in out: - return False, 'java11 is not installed' - KsanGwBinPath = "%s/%s" % (KsanBinPath, KsanGwBinaryName) - if not os.path.exists(KsanGwBinPath): - return False, '%s is not found' % KsanGwBinPath - - return True, '' - - def Start(self): - ret, errmsg = self.PreRequest() - if ret is False: - return ret, errmsg - Ret, Pid = IsDaemonRunningWithSystemd('KsanRecovery') - if Ret is True: - print('already running') - return False, 'ksanRecovery is already running' - else: - StartCmd = 'cd %s; nohup java -jar -Dlogback.configurationFile=%s %s >/dev/null 2>&1 &' % \ - (KsanBinPath, GwXmlFilePath, KsanGwBinaryName) - os.system(StartCmd) - time.sleep(2) - - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - return True, '' - else: - return False, 'Fail to start ksanGw' - - def Stop(self): - Ret, Pid = IsDaemonRunningWithSystemd(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is False: - return False, 'ksanGw is not running' - else: - - try: - os.kill(int(Pid), signal.SIGTERM) - os.unlink(KsanGwPidFile) - print('Done') - return True, '' - except OSError as err: - return False, 'Fail to stop. %s' % err - - def Restart(self): - Ret, ErrMsg = self.Stop() - if Ret is not True: - print(ErrMsg) - Ret, ErrMsg = self.Start() - if Ret is not True: - print(ErrMsg) - else: - print('Done') - - - def Status(self): - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine=KsanGwBinaryName) - if Ret is True: - print('KsanGw ... Ok') - return True, 'KsanGw ... Ok' - else: - print('KsanGw ... Not Ok') - return False, 'KsanGw ... Not Ok' diff --git a/core/common/service/service_api.py b/core/common/service/service_api.py new file mode 100644 index 00000000..f7c28358 --- /dev/null +++ b/core/common/service/service_api.py @@ -0,0 +1,1399 @@ +#!/usr/bin/env python3 +""" +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +""" + +import sys, os +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) + +from portal_api.apis import * +from const.service import AddServiceInfoObject, \ + UpdateServiceInfoObject, UpdateServiceUsageObject, \ + AddServiceGroupItems +from const.common import ServiceGroupItemsModule, ServiceDetailModule, ServiceItemsDetailModule +from service.control import ServiceUnit + +ConfigMaxWidthLen = 100 +ConfigMaxValueLen = 100 - 13 +ConfigCompartLine = '%s' % ('=' * ConfigMaxWidthLen) +ConfigTitleLine = '%s' % ('-' * ConfigMaxWidthLen) + + +OsdDefaultPoolsize = 650 +OsdDefaultPort = 8000 +OsdDefaultEcScheduleMinute = 30000 +OsdDefaultEcApplyMinute = 3000000 +OsdDefaultEcFileSize = 100000 +OsdDefaultCacheDisk = '' +OsdDefaultCacheScheduleMinute = 2 +OsdDefaultCacheFileSize = 1024 +OsdDefaultCacheLimitMinute = 5 +OsdDefaultTrashScheduleMinute = 5 +OsdDefaultPerformanceMode = 'NO_OPTION' + +OsdDefaultConfigInfo = [{'key': 'osd.pool_size', 'value': OsdDefaultPoolsize, 'type': int, 'question': 'Insert connection pool size'}, + {'key': 'osd.port', 'value': OsdDefaultPort, 'type': int, 'question': 'Insert ksanOSD port'}, + {'key': 'osd.ec_check_interval', 'value': OsdDefaultEcScheduleMinute, 'type': int, 'question': 'Insert EC check interval(ms)'}, + {'key': 'osd.ec_wait_time', 'value': OsdDefaultEcApplyMinute, 'type': int, 'question': 'Insert EC wait time(ms)'}, + {'key': 'osd.ec_min_size', 'value': OsdDefaultEcFileSize, 'type': int, 'question': 'Insert EC file size'}, + {'key': 'osd.cache_diskpath', 'value': OsdDefaultCacheDisk, 'type': str, 'question': 'Insert cache disk path', 'value_command': 'disable: NULL, '}, + {'key': 'osd.cache_check_interval', 'value': OsdDefaultCacheScheduleMinute, 'type': int, 'question': 'Insert cache check interval(ms)'}, + {'key': 'osd.cache_expire', 'value': OsdDefaultCacheLimitMinute, 'type': int, 'question': 'Insert cache expire(ms)'}, + {'key': 'osd.trash_check_interval', 'value': OsdDefaultTrashScheduleMinute, 'type': int, 'question': 'Insert trash check interval(ms)'}] + + +S3DefaultDbRepository = TypeServiceMariaDB +S3DefaultDbHost = '127.0.0.1' +S3DefaultDatabase = 'ksan' +S3DefaultDbPort = 3306 +S3DefaultDbUser = 'root' +S3DefaultDbPassword = 'YOUR_DB_PASSWORD' +S3DefaultDbPoolSize = 20 +S3DefaultGwAuthrization = 'AWS_V2_OR_V4' +S3DefaultGwEndpoint = 'http://0.0.0.0:8080' +S3DefaultGwSecureEndpoint = 'https://0.0.0.0:8443' +S3DefaultGwKeyStorePath = '/usr/local/ksan/ssl/pspace.jks' +S3DefaultGwtKeyStorePassword = 'YOUR_JKS_PASSWORD' +S3DefaultGwMaxFileSize = 3221225472 +S3DefaultGwMaxListSize = 200000 +S3DefaultGwMaxTimeSkew = 9000 +S3DefaultGwLogging = 'on' +S3DefaultGwReplication = 2 +S3DefaultGwOsdPort = 8000 +S3DefaultGwJettyMaxThreads = 1000 +S3DefaultGwJettyMaxIdleTime = 600000 +S3DefaultGwOsdClientCount = 100 +S3DefaultGwObjmanagerCount = 100 +S3DefaultGwPerformanceMode = 'NO_OPTION' +S3DefaultCacheDisk = '' +S3DefaultCacheFileSize = 1024 + +ObjManagerDefaultDbRepository = TypeServiceMariaDB +ObjManagerDefaultDbHost = '127.0.0.1' +ObjManagerDefaultDatabase = 'ksan' +ObjManagerDefaultDbPort = 3306 +ObjManagerDefaultDbUser = 'root' +ObjManagerDefaultDbPassword = 'YOUR_DB_PASSWORD' +ObjManagerDefaultMqHost = '127.0.0.1' +ObjManagerDefaultMqName = 'disk' +ObjManagerDefaultMqExchangeName = 'ksan.system' +ObjManagerDefaultMqOsdExchangeName = 'OSDExchange' + +S3DefaultConfigInfo = [ + #{'key': 'gw.db_repository', 'value': S3DefaultDbRepository, 'type': str, 'question': 'Insert ksanGW DB repository', 'valid_answer_list': [TypeServiceMariaDB, TypeServiceMongoDB]}, + #{'key': 'gw.db_host', 'value': S3DefaultDbHost, 'type': str, 'question': 'Insert ksanGW DB host'}, + #{'key': 'gw.db_name', 'value': S3DefaultDatabase, 'type': str, 'question': 'Insert ksanGW DB name'}, + #{'key': 'gw.db_port', 'value': S3DefaultDbPort, 'type': int, 'question': 'Insert ksanGW DB port'}, + #{'key': 'gw.db_user', 'value': S3DefaultDbUser, 'type': str, 'question': 'Insert ksanGW DB user'}, + #{'key': 'gw.db_password', 'value': S3DefaultDbPassword, 'type': str, 'question': 'Insert ksanGW DB password'}, + #{'key': 'gw.db_pool_size', 'value': S3DefaultDbPoolSize, 'type': int, 'question': 'Insert ksanGW DB pool size'}, + {'key': 'gw.authorization', 'value': S3DefaultGwAuthrization, 'type': str, 'question': 'Insert authorization'}, + {'key': 'gw.endpoint', 'value': S3DefaultGwEndpoint, 'type': 'url', 'question': 'Insert endpoint url'}, + {'key': 'gw.secure_endpoint', 'value': S3DefaultGwSecureEndpoint, 'type': 'url', + 'question': 'Insert secure endpoint url'}, + {'key': 'gw.keystore_path', 'value': S3DefaultGwKeyStorePath, 'type': str, 'question': 'Insert keystore path'}, + {'key': 'gw.keystore_password', 'value': S3DefaultGwtKeyStorePassword, 'type': str, + 'question': 'Insert keystore password'}, + {'key': 'gw.max_file_size', 'value': S3DefaultGwMaxFileSize, 'type': int, 'question': 'Insert max file size'}, + {'key': 'gw.max_list_size', 'value': S3DefaultGwMaxListSize, 'type': int, 'question': 'Insert max sist size'}, + {'key': 'gw.max_timeskew', 'value': S3DefaultGwMaxTimeSkew, 'type': int, 'question': 'Insert max time skew'}, + {'key': 'gw.logging', 'value': S3DefaultGwLogging, 'type': str, 'question': 'Insert logging on/off', 'valid_answer_list': ['on', 'off']}, + {'key': 'gw.osd_port', 'value': S3DefaultGwOsdPort, 'type': int, 'question': 'Insert ksanOSD port'}, + {'key': 'gw.jetty_max_threads', 'value': S3DefaultGwJettyMaxThreads, 'type': int, + 'question': 'Insert Jetty max thread count'}, + {'key': 'gw.jetty_max_idle_timeout', 'value': S3DefaultGwJettyMaxIdleTime, 'type': int, + 'question': 'Insert Jetty max idle time'}, + {'key': 'gw.osd_client_count', 'value': S3DefaultGwOsdClientCount, 'type': int, + 'question': 'Insert ksanOSD client count'}, + {'key': 'gw.objmanager_count', 'value': S3DefaultGwObjmanagerCount, 'type': int, + 'question': 'Insert ksanObjManager count'}, + {'key': 'gw.performance_mode', 'value': S3DefaultGwPerformanceMode, 'type': str, + 'question': 'Insert performance mode'}, + {'key': 'gw.cache_diskpath', 'value': S3DefaultCacheDisk, 'type': str, 'question': 'Insert cache disk path', 'value_command': 'disable: NULL, '}, + {'key': 'gw.cache_file_size', 'value': S3DefaultCacheFileSize, 'type': int, 'question': 'Insert cache file size'}, + {'key': 'objM.db_repository', 'value': ObjManagerDefaultDbRepository, 'type': str, + 'question': 'Insert object DB repository', 'valid_answer_list': [TypeServiceMariaDB, TypeServiceMongoDB]}, + #{'key': 'objM.db_host', 'value': ObjManagerDefaultDbHost, 'type': str, 'question': 'Insert object DB host', 'valid_answer_list': [TypeServiceMariaDB, TypeServiceMongoDB]}, + {'key': 'objM.db_host', 'value': ObjManagerDefaultDbHost, 'type': str, 'question': 'Insert object DB host'}, + {'key': 'objM.db_name', 'value': ObjManagerDefaultDatabase, 'type': str, 'question': 'Insert object DB name'}, + {'key': 'objM.db_port', 'value': ObjManagerDefaultDbPort, 'type': int, 'question': 'Insert object DB port'}, + {'key': 'objM.db_user', 'value': ObjManagerDefaultDbUser, 'type': str, 'question': 'Insert object DB user'}, + {'key': 'objM.db_password', 'value': ObjManagerDefaultDbPassword, 'type': str, 'question': 'Insert object DB password'} + #{'key': 'objM.mq_host', 'value': ObjManagerDefaultMqHost, 'type': str, 'question': 'Insert MQ host', 'Activate': False}, + #{'key': 'objM.mq_queue_name', 'value': ObjManagerDefaultMqName, 'type': str, 'question': 'Insert MQ name', 'Activate': False}, + #{'key': 'objM.mq_exchange_name', 'value': ObjManagerDefaultMqExchangeName, 'type': str, 'question': 'Insert MQ exchange name', 'Activate': False}, + #{'key': 'objM.mq_osd_exchange_name', 'value': ObjManagerDefaultMqOsdExchangeName, 'type': str, 'question': 'Insert MQ ksanOSD exchange name', 'Activate': False} + #{'key': 'objM.mq.host', 'value': ObjManagerDefaultMqHost, 'type': 'ip', 'question': 'Insert MQ Host'}, + #{'key': 'objM.mq.diskpool.queuename', 'value': ObjManagerDefaultMqDiskpoolQueueName, 'type': str, + # 'question': 'Insert MQ Disk pool Queue Name'}, + #{'key': 'objM.mq.diskpool.exchangename', 'value': ObjManagerDefaultMqDiskpoolExchangeName, 'type': str, + # 'question': 'Insert MQ Disk pool Exchange Name'}, + #{'key': 'objM.mq.osd.exchangename=OSDExchange', 'value': ObjManagerDefaultMqOsdExchangeName, 'type': str, + # 'question': 'Insert MQ ksanOSD Exchange Name'} +] + + + + + + + + + + +MongoDbDefaultShard1Port = 20001 +MongoDbDefaultShard2Port = 20002 +MongoDbDefaultConfigServerPort = 50000 +MongoDbDefaultDbPort = 27017 +MongoDbDefaultHomeDir = '/var/lib/mongo' +MongoDbDefaultPrimaryHost = socket.gethostname() + +MongoDbConfigInfo = [ + {'key': 'Shard1Port', 'value': MongoDbDefaultShard1Port, 'type': int, 'question': 'Insert MongoDB shard1 port'}, + {'key': 'Shard2Port', 'value': MongoDbDefaultShard2Port, 'type': int, 'question': 'Insert MongoDB shard2 port'}, + {'key': 'ConfigServerPort', 'value': MongoDbDefaultConfigServerPort, 'type': int, 'question': 'Insert MongoDB config server port'}, + {'key': 'MongoDbPort', 'value': MongoDbDefaultDbPort, 'type': int, + 'question': 'Insert MongoDB DB port'}, + {'key': 'HomeDir', 'value': MongoDbDefaultHomeDir, 'type': str, + 'question': 'Insert MongoDB home directory'}, + {'key': 'PrimaryHostName', 'value': MongoDbDefaultPrimaryHost, 'type': str, + 'question': 'Insert MongoDB primary cluster node hostname'} +] + +lifecycleDefaultDbRepository = TypeServiceMariaDB +lifecycleDefaultDbHost = '127.0.0.1' +lifecycleDefaultDbPort = 3306 +lifecycleDefaultDbName = 'ksan' +lifecycleDefaultDbUser = 'root' +lifecycleDefaultDbPassword = 'YOUR_DB_PASSWORD' +lifecycleDefaultRegion = 'kr-ksan-1' +lifecycleDefaultRuntime = '01:00' +lifecycleDefaultCheckInterval = 5000 + +LifecycleDefaultConfigInfo = [{'key': 'objM.db_repository', 'value': lifecycleDefaultDbRepository, 'type': str, 'question': 'Insert ksanGW DB repository'}, + {'key': 'objM.db_host', 'value': lifecycleDefaultDbHost, 'type': str, 'question': 'Insert ksanGW DB host'}, + {'key': 'objM.db_port', 'value': lifecycleDefaultDbPort, 'type': int, 'question': 'Insert ksanGW DB port'}, + {'key': 'objM.db_name', 'value': lifecycleDefaultDbName, 'type': str, 'question': 'Insert ksanGW DB name'}, + {'key': 'objM.db_user', 'value': lifecycleDefaultDbUser, 'type': str, 'question': 'Insert ksanGW DB user'}, + {'key': 'objM.db_password', 'value': lifecycleDefaultDbPassword, 'type': str, 'question': 'Insert ksanGW DB password'}, + {'key': 'ksan.region', 'value': lifecycleDefaultRegion, 'type': str, 'question': 'Insert region'}, + {'key': 'lifecycle.schedule', 'value': lifecycleDefaultRuntime, 'type': str, 'question': 'Insert runtime'}, + {'key': 'lifecycle.check_interval', 'value': lifecycleDefaultCheckInterval, 'type': int, 'question': 'Insert check interval'}] + + +ReplicationDefaultUploadThreadCount = 20 +ReplicationDefaultMultipartSize = 5242880 +ReplicationDefaultConfigInfo = [ + + {'key': 'objM.db_repository', 'value': S3DefaultDbRepository, 'type': str, 'question': 'Insert object DB repository', 'valid_answer_list': [TypeServiceMariaDB, TypeServiceMongoDB]}, + {'key': 'objM.db_host', 'value': ObjManagerDefaultDbHost, 'type': str, 'question': 'Insert object DB host'}, + {'key': 'objM.db_port', 'value': ObjManagerDefaultDbPort, 'type': int, 'question': 'Insert object DB port'}, + {'key': 'objM.db_name', 'value': ObjManagerDefaultDatabase, 'type': str, 'question': 'Insert object DB name'}, + {'key': 'objM.db_user', 'value': ObjManagerDefaultDbUser, 'type': str, 'question': 'Insert object DB user'}, + {'key': 'objM.db_password', 'value': ObjManagerDefaultDbPassword, 'type': str, + 'question': 'Insert object DB password'}, + {'key': 'ksan.region', 'value': lifecycleDefaultRegion, 'type': str, 'question': 'Insert region'}, + {'key': 'replication.upload_thread_count', 'value': ReplicationDefaultUploadThreadCount, 'type': int, 'question': 'Insert upload thread count'}, + {'key': 'replication.multipart_size', 'value': ReplicationDefaultMultipartSize, 'type': int, + 'question': 'Insert multipart size'} +] + +DefaultDbLogExpireDays = 30 +LogManagerDefaultCheckInterval = 300000 +LogManagerDefaultConfigInfo = [ + {'key': 'ksan.region', 'value': lifecycleDefaultRegion, 'type': str, 'question': 'Insert region'}, + {'key': 'logM.db_repository', 'value': S3DefaultDbRepository, 'type': str, + 'question': 'Insert object DB repository', + 'valid_answer_list': [TypeServiceMariaDB, TypeServiceMongoDB]}, + {'key': 'logM.db_host', 'value': ObjManagerDefaultDbHost, 'type': str, + 'question': 'Insert object DB host'}, + {'key': 'logM.db_port', 'value': ObjManagerDefaultDbPort, 'type': int, 'question': 'Insert object DB port'}, + {'key': 'logM.db_name', 'value': ObjManagerDefaultDatabase, 'type': str, 'question': 'Insert object DB name'}, + {'key': 'logM.db_user', 'value': ObjManagerDefaultDbUser, 'type': str, + 'question': 'Insert object DB user'}, + {'key': 'logM.db_password', 'value': ObjManagerDefaultDbPassword, 'type': str, + 'question': 'Insert object DB password'}, + {'key': 'logM.db_pool_size', 'value': S3DefaultDbPoolSize, 'type': str, 'question': 'Insert object DB pool size'}, + {'key': 'logM.db_expires', 'value': DefaultDbLogExpireDays, 'type': str, 'question': 'Insert object DB log expire days'}, + {'key': 'logM.check_interval', 'value': LogManagerDefaultCheckInterval, 'type': str, 'question': 'Insert check interval'}, +] + + +BackendConfigInfo = dict() +BackendConfigInfo[TypeServiceLifecycle] = LifecycleDefaultConfigInfo +BackendConfigInfo[TypeServiceReplication] = ReplicationDefaultConfigInfo +BackendConfigInfo[TypeServiceLogManager] = LogManagerDefaultConfigInfo + +@catch_exceptions() +def AddService(MgsIp, Port, ApiKey, ServiceName, ServiceType, ServerId=None, VlanIds=[], ServerName=None, GroupId='', Description='', logger=None): + #VlanIds = GetVlanIdListFromServerDetail(MgsIp, Port, ApiKey, ServerId=ServerId, ServerName=ServerName, logger=logger) + #if VlanIds is None: + # return ResEtcErrorCode, ResFailToGetVlainId, None + if ServerId is not None: + TargetServer = ServerId + elif ServerName is not None: + TargetServer = ServerName + else: + return ResInvalidCode, ResInvalidMsg + 'ServerName is required', None + + + Service = AddServiceInfoObject() + Service.Set(ServiceName, TargetServer, ServiceType, GroupId, VlanIds, State='Offline', Description=Description, HaAction='Initializing') + if ServiceType is None: + return ResInvalidCode, ResInvalidMsg + 'ServiceType is required', None + + Url = "/api/v1/Services" + body = jsonpickle.encode(Service, make_refs=False) + Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=body, logger=logger) + Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + +def RegisterService(conf, ServiceType, logger, ServiceNicname=None): + """ + + :param Conf: + :param ServiceType: + :param logger: + :return: + """ + out, err = shcall('hostname') + ServiceName = '%s_%s' % (ServiceNicname, out[:-1]) + Res, Errmsg, Ret = AddService(conf.PortalHost, conf.PortalPort, conf.PortalApiKey, ServiceName, ServiceType, + conf.ServerId, logger=logger) + if Res == ResOk: + if Ret.Result != ResultSuccess: + if Ret.Result != CodeDuplicated: + logger.error("%s %s" % (str(Ret.Result), str(Ret.Message))) + else: + logging.log(logging.INFO, "%s is registered. %s %s" % (ServiceType, str(Ret.Result), str(Ret.Message))) + else: + logger.error("%s %s" % (str(Ret.Result), str(Ret.Message))) + +def KsanServiceRegister(conf, ServiceType, logger): + Retry = 0 + while True: + Retry += 1 + Res, Errmsg, Ret, Data = GetServerInfo(conf.PortalHost, conf.PortalPort,conf.PortalApiKey , ServerId=conf.ServerId, logger=logger) + if Res == ResOk: + if Ret.Result == ResultSuccess: + NetworkInterfaces = Data.NetworkInterfaces + if len(NetworkInterfaces) > 0: + isNetworkAdded = True + break + else: + logger.error('fail to get network interface %s' % Ret.Message) + else: + logger.error('fail to get network interface %s' % Errmsg) + if Retry > ServiceContolRetryCount: + isNetworkAdded = False + break + time.sleep(IntervalMiddle) + + if isNetworkAdded is True: + RegisterService(conf, ServiceType, logger) + +@catch_exceptions() +def DeleteService(ip, port, ApiKey, ServiceId=None, ServiceName=None, logger=None): + """ + delete serviceinfo from server pool + :param ip: + :param port: + :param Id: + :param logger: + :return:tuple(error code, error msg, ResponseHeader class) + """ + if ServiceId is not None: + TargetServcie = ServiceId + elif ServiceName is not None: + TargetServcie = ServiceName + else: + return ResInvalidCode, ResInvalidMsg, None + + Url = '/api/v1/Services/%s' % TargetServcie + Conn = RestApi(ip, port, Url, authkey=ApiKey, logger=logger) + Res, Errmsg, Ret = Conn.delete() + return Res, Errmsg, Ret + + +@catch_exceptions() +def GetServiceInfo(MgsIp, Port, ApiKey, ServiceId=None, ServiceName=None, logger=None): + """ + get service info all or specific service info with Id + :param ip: + :param port: + :param disp: + :return: + """ + if ServiceId is not None: + TargetService = ServiceId + elif ServiceName is not None: + TargetService = ServiceName + else: + TargetService = None + + if TargetService is not None: + Url = "/api/v1/Services/%s" % TargetService + ItemsHeader = False + ReturnType = ServiceDetailModule + else: + Url = "/api/v1/Services/" + ItemsHeader = True + ReturnType = ServiceItemsDetailModule + Params = dict() + Params['countPerPage'] = 100 + Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=Params, logger=logger) + Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) + if Res == ResOk: + AllServiceInfo = list() + if Ret.Result == ResultSuccess: + if TargetService is not None: + return Res, Errmsg, Ret, Ret.Data + else: + return Res, Errmsg, Ret, Ret.Data.Items + return Res, Errmsg, Ret, None + else: + return Res, Errmsg, None, None + +@catch_exceptions() +def GetServiceMongoDBConfig(MgsIp, Port, ApiKey, logger=None): + """ + get service info all or specific service info with Id + :param ip: + :param port: + :param disp: + :return: + """ + Url = "/api/v1/Config/MongoDB" + ItemsHeader = False + ReturnType = ServiceConfigModule + + Params = dict() + Params['countPerPage'] = 100 + Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=Params, logger=logger) + Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) + if Res == ResOk: + AllServiceInfo = list() + if Ret.Result == ResultSuccess: + return Res, Errmsg, Ret, Ret.Data + return Res, Errmsg, Ret, None + else: + return Res, Errmsg, None, None + + +@catch_exceptions() +def UpdateServiceInfo(MgsIp, Port, ApiKey, ServiceId=None, ServiceName=None, GroupId=None, Description=None, ServiceType=None, + HaAction=None, State=None, logger=None): + Res, Errmsg, Ret, Service = GetServiceInfo(MgsIp, Port, ApiKey, ServiceId=ServiceId, ServiceName=ServiceName, logger=logger) + if Res != ResOk: + return Res, Errmsg, None + else: + if Ret.Result != ResultSuccess: + return Res, Errmsg, Ret + + VlanIds = GetVlanIdListFromVlanList(Service.Vlans) + NewService = UpdateServiceInfoObject() + NewService.Set(Service.Name, Service.ServiceType, Service.GroupId, VlanIds, Service.State, + Service.Description, Service.HaAction) + + if ServiceName is not None: + NewService.Name = ServiceName + if GroupId is not None: + NewService.GroupId = GroupId + if Description is not None: + NewService.Description = Description + if ServiceType is not None: + NewService.ServiceType = ServiceType + if HaAction is not None: + NewService.HaAction = HaAction + if VlanIds is not None: + NewService.VlanIds = VlanIds + if State is not None: + NewService.State = State + + if ServiceId is not None: + TargetServcie = ServiceId + elif ServiceName is not None: + TargetServcie = ServiceName + else: + return ResInvalidCode, ResInvalidMsg, None + + Url = "/api/v1/Services/%s" % TargetServcie + body = jsonpickle.encode(NewService, make_refs=False) + Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=body, logger=logger) + Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + + + +@catch_exceptions() +def UpdateServiceState(MgsIp, Port, ApiKey, ServiceId, State, logger=None): + + Url = "/api/v1/Services/%s/State/%s" % (ServiceId, State) + body = dict() + Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=body, logger=logger) + Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + + +@catch_exceptions() +def UpdateServiceUsage(MgsIp, Port, ApiKey, ServiceId, logger=None): + Usage = UpdateServiceUsageObject() + Usage.Set(ServiceId, 48, 55555, 88) + Url = "/api/v1/Services/Usage" + body = jsonpickle.encode(Usage, make_refs=False) + Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=body, logger=logger) + Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + + +@catch_exceptions() +def ControlService(MgsIp, Port, ApiKey, Control, ServiceId=None, ServiceName=None, logger=None): + Res, Errmsg, Ret, Data = GetServiceInfo(MgsIp, Port, ApiKey, ServiceId=ServiceId, ServiceName=ServiceName, logger=logger) + if Res != ResOk: + return Res, Errmsg, None + else: + if Ret.Result != ResultSuccess: + return Res, Errmsg, Ret + + if ServiceId is not None: + TargetServcie = ServiceId + elif ServiceName is not None: + TargetServcie = ServiceName + else: + return ResInvalidCode, ResInvalidMsg, None + + Url = "/api/v1/Services/%s/%s" % (TargetServcie, Control) + body = dict() + Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey,params=body, logger=logger) + Res, Errmsg, Data = Conn.post() + return Res, Errmsg, Data + + +def TmpGetServiceInfoList(Ip, Port): + Res, Errmsg, Ret = GetServiceInfo(Ip, Port) + if Res == ResOk: + if Ret == ResultSuccess: + return Res, Errmsg, Ret.Data + else: + return Ret.Result, Ret.Message, None + return Res, Errmsg, None + + +@catch_exceptions() +def ShowServiceInfoWithServerInfo(ServiceList, Detail=DetailInfo, Ip=None, Port=None, SysinfoDisp=False): + #Ret, Errmsg , Data = TmpGetServiceInfoList(Ip, Port) + #if Ret != ResOk: + # print('Fail to get Service Info %s' % Errmsg) + # return + + ServiceList = ServiceListOrdering(ServiceList, OrderingBase='ServerName') + if SysinfoDisp is True: + ServiceTitleLine = '%s' % ('=' * 105) + ServiceTitle ="|%s|%s|%s|%s|%s|" % ('ServerName'.ljust(25), 'ServiceName'.ljust(30), 'Status'.ljust(10), 'Type'.ljust(30), ' ' * 4) + ServiceDataLine = '%s' % ('-' * 105) + else: + if Detail == MoreDetailInfo: + + ServiceTitleLine = '%s' % ('=' * 163) + ServiceDataLine = '%s' % ('-' * 163) + ServiceTitle = "|%s|%s|%s|%s|%s|%s|%s|%s|" % ( + 'ServerName'.ljust(20), 'ServiceName'.ljust(30), 'Status'.ljust(10), + 'Type'.ljust(30), 'CpuUsage'.ljust(8), 'MemUsed'.ljust(8), 'Thread Cnt'.ljust(10), + 'ServiceId'.ljust(38)) + elif Detail == DetailInfo: + ServiceTitleLine = '%s' % ('=' * 124) + ServiceDataLine = '%s' % ('-' * 124) + ServiceTitle = "|%s|%s|%s|%s|%s|%s|%s|" % ( + 'ServerName'.ljust(20), 'ServiceName'.ljust(30), + 'Status'.ljust(10), 'Type'.ljust(30), 'CpuUsage'.ljust(8), 'MemUsed'.ljust(8), + 'Thread Cnt'.ljust(10)) + else: + ServiceTitleLine = '%s' % ('=' * 95) + ServiceDataLine = '%s' % ('-' * 95) + ServiceTitle = "|%s|%s|%s|%s|" % ('ServerName'.ljust(20), 'ServiceName'.ljust(30), + 'Status'.ljust(10), 'Type'.ljust(30)) + + print(ServiceTitleLine) + print(ServiceTitle) + print(ServiceTitleLine) + + if len(ServiceList) <= 0: + print('No service data') + print(ServiceDataLine) + else: + for Svc in ServiceList: + if SysinfoDisp is True: + _Svc = "|%s|%s|%s|%s|%s|" % (Svc.Server.Name.ljust(25), str(Svc.Name).ljust(30), DisplayState(Svc.State).ljust(10), str(Svc.ServiceType).ljust(30), ' ' * 4) + else: + if Detail == MoreDetailInfo: + _Svc = "|%s|%s|%s|%s|%s|%s|%s|%s|" % \ + ( Svc.Server.Name.ljust(20), str(Svc.Name).ljust(30), + DisplayState(Svc.State).ljust(10), str(Svc.ServiceType).ljust(30), + "{:,}".format(Svc.CpuUsage).rjust(8), + Byte2HumanValue(str(Svc.MemoryUsed), 'UsedSize').rjust(10), "{:,}".format(int(Svc.ThreadCount)).rjust(10), Svc.Id.ljust(38)) + elif Detail == DetailInfo: + _Svc = "|%s|%s|%s|%s|%s|%s|%s|" % \ + (Svc.Server.Name.ljust(20), str(Svc.Name).ljust(30), + DisplayState(Svc.State).ljust(10), str(Svc.ServiceType).ljust(30), + "{:,}".format(Svc.CpuUsage).rjust(8), + Byte2HumanValue(str(float(Svc.MemoryUsed)), 'UsedSize').rjust(10), "{:,}".format(int(Svc.ThreadCount)).rjust(10)) + else: + _Svc ="|%s|%s|%s|%s|" % \ + (Svc.Server.Name.ljust(20), str(Svc.Name).ljust(30), DisplayState(Svc.State).ljust(10), str(Svc.ServiceType).ljust(30)) + + print(_Svc) + print(ServiceDataLine) + +def ServiceListOrdering(ServiceList, OrderingBase='ServerName'): + TotalNewServiceList = list() + + if OrderingBase == 'ServerName': + ServerNameDict = dict() + for serviceinfo in ServiceList: + ServerName = serviceinfo.Server.Name + if ServerName not in ServerNameDict: + ServerNameDict[ServerName] = list() + ServerNameDict[ServerName].append(serviceinfo) + else: + ServerNameDict[ServerName].append(serviceinfo) + + for servername in sorted(ServerNameDict.keys(), key=str.casefold): + servicelist = ServerNameDict[servername] + ServiceNameDict = dict() + for idx, service in enumerate(servicelist): + ServiceNameDict[service.Name] = service + + newservicelist = list() + for servicename in sorted(ServiceNameDict.keys(), key=str.casefold) : + service = ServiceNameDict[servicename] + newservicelist.append(service) + + TotalNewServiceList += newservicelist + else: + ServiceNameDict = dict() + for serviceinfo in ServiceList: + servicename = serviceinfo.Name + ServiceNameDict[servicename] = serviceinfo + + for servicename in sorted(ServiceNameDict.keys(), key=str.casefold): + serviceinfo = ServiceNameDict[servicename] + TotalNewServiceList.append(serviceinfo) + + + return TotalNewServiceList + + + +@catch_exceptions() +def GetVlanIdListFromServerDetail(MgsIp, PortarPort, ApiKey, ServerId=None, ServerName=None, logger=None): + retry = 0 + VlanIds = list() + while True: + retry += 1 + Res, Errmsg, Ret, Data = GetServerInfo(MgsIp, PortarPort, ApiKey, ServerId=ServerId, Name=ServerName, logger=logger) + if Res == ResOk: + if Ret.Result == ResultSuccess: + for net in Data.NetworkInterfaces: + for vlan in net.NetworkInterfaceVlans: + VlanIds.append(vlan.Id) + break + else: + if retry > 3: + break + else: + time.sleep(1) + return VlanIds + + +@catch_exceptions() +def GetVlanIdListFromVlanList(VlanList): + VlanIds = list() + for Vlan in VlanList: + VlanIds.append(Vlan.Id) + return VlanIds + + +@catch_exceptions() +def AddServiceGroup(Ip, Port, Name, ServiceType, ServiceIpAddress=None, ServiceIds=[], Description='', logger=None): + + Group = AddServiceGroupItems() + Group.Set(Name, ServiceType, ServiceIpAddress=ServiceIpAddress, ServiceIds=ServiceIds) + + Url = "/api/v1/ServiceGroups" + body = jsonpickle.encode(Group, make_refs=False) + Conn = RestApi(Ip, Port, Url, params=body, logger=logger) + Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + + +@catch_exceptions() +def RemoveServiceGroup(ip, port, ServiceId, logger=None): + """ + delete serviceinfo from server pool + :param ip: + :param port: + :param Id: + :param logger: + :return:tuple(error code, error msg, ResponseHeader class) + """ + Url = '/api/v1/ServiceGroups/%s' % ServiceId + Conn = RestApi(ip, port, Url, logger=logger) + Res, Errmsg, Ret = Conn.delete() + return Res, Errmsg, Ret + + +@catch_exceptions() +def UpdateServiceGroup(Ip, Port, GroupId, Name=None, Description=None, ServiceType=None, + ServiceIds=None, addServiceIds=None, removeServiceIds=None, logger=None): + Res, Errmsg, Ret,Group = GetServiceGroup(Ip, Port, GroupId=GroupId, logger=logger) + if Res != ResOk: + return Res, Errmsg, None + else: + if Ret.Result != ResultSuccess: + return Res, Errmsg, Ret + + ExistServiceIds = GetServiceIdList(Group) + if Name is not None: + Group.Name = Name + if Description is not None: + Group.Description = Description + if ServiceType is not None: + Group.ServiceType = ServiceType + if ServiceIds is not None: + ExistServiceIds = ServiceIds + if addServiceIds is not None: + for new in addServiceIds: + if new in ExistServiceIds: + return ResDuplicateCode, ResDuplicateMsg, None + + ExistServiceIds += addServiceIds + if removeServiceIds is not None: + for ServiceId in removeServiceIds: + ExistServiceIds.remove(ServiceId) + + NewGroup = AddServiceGroupItems() + NewGroup.Set(Group.Name, Group.ServiceType, ServiceIpAddress=Group.ServiceIpAddress, ServiceIds=ExistServiceIds) + + Url = "/api/v1/ServiceGroups/%s" % GroupId + body = jsonpickle.encode(NewGroup, make_refs=False) + Conn = RestApi(Ip, Port, Url, params=body, logger=logger) + Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + + +def ControlSerivceGroup(Ip, Port, GroupId, Action, logger=None): + Url = "/api/v1/ServiceGroups/%s/%s" % (GroupId, Action) + body = dict() + Conn = RestApi(Ip, Port, Url, params=body, logger=logger) + Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + +def StopSerivceGroup(Ip, Port, GroupId, logger=None): + Url = "/api/v1/ServiceGroups/%s/Stop" % GroupId + body = dict() + Conn = RestApi(Ip, Port, Url, params=body, logger=logger) + Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + + +def GetServiceIdList(GroupDetail): + ServiceListOfGroup = list() + for Service in GroupDetail.Services: + ServiceListOfGroup.append(Service.Id) + return ServiceListOfGroup + + +def GetServiceGroup(Ip, Port, GroupId=None, logger=None): + if GroupId is not None: + Url = "/api/v1/ServiceGroups/%s" % GroupId + ItemsHeader = False + ReturnType = ServiceGroupItemsModule + else: + Url = "/api/v1/ServiceGroups" + ItemsHeader = True + ReturnType = ResponseItemsHeaderModule + Params = dict() + Params['countPerPage'] = 100 + Conn = RestApi(Ip, Port, Url, params=Params, logger=logger) + Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) + if Res == ResOk: + AllServiceInfo = list() + if Ret.Result == ResultSuccess: + if GroupId is not None: + #GetDataFromBodyReculsive(Ret.Data, ServiceDetailModule) + #ServiceDetailInfo = jsonpickle.decode(json.dumps(Ret.Data)) + return Res, Errmsg, Ret, Ret.Data + else: + return Res, Errmsg, Ret, Ret.Data.Items + return Res, Errmsg, Ret, None + else: + return Res, Errmsg, None, None + + +def GetAllServiceGroups(Ip, Port, logger=None): + Res, Errmsg, Ret, Groups = GetServiceGroup(Ip, Port, logger=logger) + if Res == ResOk: + AllServiceGroupList = list() + if Ret.Result == ResultSuccess: + for Grp in Groups: + res, errmsg, ret, GroupDetail = GetServiceGroup(Ip, Port, GroupId=Grp.Id, logger=logger) + if res == ResOk: + if ret.Result == ResultSuccess: + AllServiceGroupList.append(GroupDetail) + else: + print('fail to get Service Group Info', ret.Message) + return Res, Errmsg, Ret, AllServiceGroupList + else: + print('fail to get Service Group List', Ret.Message) + return Res, Errmsg, Ret, None + else: + return Res, Errmsg, None, None + + +@catch_exceptions() +def ShowServiceGroup(Groups, Detail=False): + + TopTitleLine = '%s' % ('=' * 175) + GroupTitleLine = '%s' % ('-' * 175) + print(TopTitleLine) + GroupTitle ="|%s|%s|%s|%s|%s|%s|" % ('Name'.center(14), 'Description'.center(20), 'Id'.center(40), + 'ServiceType'.center(15), 'ServiceIpaddress'.center(20), ' ' * 59) + print(GroupTitle) + print(TopTitleLine) + + ServiceTitleLine = '%s%s' % (' ' *10, '-' * 165) + for Grp in Groups: + _grp = "|%s|%s|%s|%s|%s|%s|" % ( Grp.Name.center(14), str(Grp.Description).center(20), Grp.Id.center(40), + Grp.ServiceType.center(15), str(Grp.ServiceIpAddress).center(20), ' ' * 59) + + print(_grp) + print(GroupTitleLine) + if Detail: + ServiceTitle = "%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 10, 'Service Name'.center(16), 'Description'.center(20), + 'Service Id'.center(40), 'State'.center(15), 'Service Type'.center(15), + 'Cpu Usage'.center(20), 'Memoy Used'.center(20), 'Thread Cnt'.center(10)) + print(ServiceTitle) + if len(Grp.Services) >= 0: + print(ServiceTitleLine) + else: + print(GroupTitleLine) + for idx, svc in enumerate(Grp.Services): + _svc ="%s|%s|%s|%s|%s|%s|%s|%s|%s|" % \ + (' ' * 10, svc.Name.center(16), '{:20.20}'.format(str(svc.Description).center(20)), + svc.Id.center(40), svc.State.center(15), str(svc.ServiceType).center(15), + str(svc.CpuUsage).center(20), str(svc.MemoryUsed).center(20), str(svc.ThreadCount).center(10)) + + print(_svc) + if len(Grp.Services) -1 == idx: + print(GroupTitleLine) + else: + print(ServiceTitleLine) + + +@catch_exceptions() +def GetSeviceConfStringForDisplay(Conf, Prefix, ConfFile=None): + Conf = json.loads(Conf, strict=False) + StrConf = '' + if ConfFile is None: + for key1, val1 in Conf.items(): + if isinstance(val1, dict): + for key2, val2 in val1.items(): + ConfLineStr = "%s_%s=%s" % (key1, key2, str(val2)) + StrConf += "%s%s|\n" % (Prefix, ConfLineStr.ljust(100-13)) + else: + ConfLineStr = "%s=%s" % (key1, str(val1)) + StrConf += '%s%s|\n' % (Prefix, ConfLineStr.ljust(100-13)) + else: + for key1, val1 in Conf.items(): + if isinstance(val1, dict): + for key2, val2 in val1.items(): + StrConf += "%s_%s=%s\n" % (key1, key2, str(val2)) + else: + StrConf += "%s=%s\n" % (key1, str(val1)) + + return StrConf + +@catch_exceptions() +def GetServiceConfString(Conf): + Conf = json.loads(Conf) + StrConf = '' + for key1, val1 in Conf.items(): + if isinstance(val1, dict): + for key2, val2 in val1.items(): + StrConf += "%s_%s=%s" % (key1, key2, str(val2)) + else: + StrConf += "%s=%s" % (key1, str(val1)) + return StrConf + + +@catch_exceptions() +def GetDspConfString(Conf, Prefix): + Conf = json.loads(Conf) + StrConf = '' + for key1, val1 in Conf.items(): + if isinstance(val1, dict): + for key2, val2 in val1.items(): + ConfLineStr = "%s_%s=%s" % (key1, key2, str(val2)) + StrConf += "%s%s|\n" % (Prefix, ConfLineStr.ljust(100-13)) + else: + ConfLineStr = "%s=%s" % (key1, str(val1)) + StrConf += '%s%s|\n' % (Prefix, ConfLineStr.ljust(100-13)) + return StrConf + + +@catch_exceptions() +def GetServiceConfigList(PortalIp, PortalPort, ApiKey, ServiceType, logger=None): + """ + Get config list from portal + :param PortalIp: + :param PortalPort: + :param ServiceType: + :param logger: + :return: + """ + Url = '/api/v1/Config/List/%s' % ServiceType + ItemsHeader = True + ReturnType = ResponseItemsHeaderModule + Params = dict() + Params['countPerPage'] = 100 + + Conn = RestApi(PortalIp, PortalPort, Url, authkey=ApiKey, params=Params, logger=logger) + Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) + if Res == ResOk: + ConfigList = list() + if Ret.Result == ResultSuccess: + return Res, Errmsg, Ret, Ret.Data + else: + return Res, Errmsg, Ret, None + else: + return Res, Errmsg, None, None + +@catch_exceptions() +def LoadServiceList(conf, ServiceList, LocalIpList, logger): + Res, Errmsg, Ret, Data = GetServerInfo(conf.PortalHost, conf.PortalPort, conf.PortalApiKey, conf.ServerId, logger=logger) + if Res == ResOk: + if Ret.Result == ResultSuccess: + for service in Data.Services: + ServiceList['IdList'].append(service.Id) + ServiceList['Details'][service.Id] = service + for Interface in Data.NetworkInterfaces: + LocalIpList.append(Interface.IpAddress) + return True + else: + logger.error('fail to get service list with serverId %s. %s' % (conf.ServerId, Ret.Message)) + return False + else: + logger.error('fail to get service list with serverId %s. %s' % (conf.ServerId, Errmsg)) + return False + +def IsValidServiceIdOfLocalServer(conf, logger, TargetServiceId): + ServiceList = dict() + LocalIpList = list() + ServiceList['IdList'] = list() + ServiceList['Details'] = dict() + RetryCnt = 3 + while True: + RetryCnt -= 1 + Ret = LoadServiceList(conf, ServiceList, LocalIpList, logger) + if Ret is True: + break + if RetryCnt < 0: + logger.error('fail to check if the added serviced %s ' % TargetServiceId) + return False + else: + time.sleep(IntervalShort) + + if TargetServiceId in ServiceList['IdList']: + return True + else: + logger.error('the added serviceId %s is not local server\'s' % TargetServiceId) + return False + +@catch_exceptions() +def ShowConfigList(ConfigList, Detail=False): + """ + Display all config info with type + :param ConfigList: + :param Detail: + :return: + """ + print(ConfigCompartLine) + for Config in ConfigList.Items: + DsPServiceConf(Config) + +def DsPServiceConf(Config, TopTitleLine=False, ConfFile=None): + """ + Display service config with type + :param Config: + :param TopTitleLine: + :return: + """ + Type = Config.Type + Version = Config.Version + Conf = Config.Config + ConfPartPrefix = '|%s|' % (' ' * 10) + RegiDate = Config.RegDate + StrConf = GetSeviceConfStringForDisplay(Conf, ConfPartPrefix, ConfFile=ConfFile) + if ConfFile is None: + if StrConf is None: + print('Invalid Conf. version:%d \n%s' % (Version, str(Conf))) + else: + #if TopTitleLine is not True: + # print(ConfigCompartLine) + ConfigStr = """%s +|%s|%s| +%s +|%s|%s| +%s +|%s|%s| +%s +|%s|%s| +%s%s """ % (ConfigCompartLine, 'Type'.center(10), Type.center(ConfigMaxValueLen), ConfigTitleLine, 'Version'.center(10), + str(Version).center(ConfigMaxValueLen), ConfigTitleLine, 'Date'.center(10), RegiDate.center(ConfigMaxValueLen), + ConfigTitleLine, + 'Conf'.center(10), ' ' * (100 - 13), StrConf.center(ConfigMaxValueLen), ConfigCompartLine) + print(ConfigStr) + else: + with open(ConfFile, 'w') as f: + f.write(StrConf) + + +@catch_exceptions() +def GetServiceConfig(PortalIp, PortalPort, ApiKey, ServiceType, Version=None, logger=None): + Url = '/api/v1/Config/%s' % ServiceType + if Version is not None: + Url += '/%s' % Version + + ItemsHeader = False + ReturnType = ServiceGroupItemsModule + + Params = dict() + Conn = RestApi(PortalIp, PortalPort, Url, authkey=ApiKey, params=Params, logger=logger) + Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) + if Res == ResOk: + AllServiceInfo = list() + if Ret.Result == ResultSuccess: + return Res, Errmsg, Ret, Ret.Data + else: + return Res, Errmsg, Ret, None + else: + return Res, Errmsg, None, None + +@catch_exceptions() +def UpdateServiceConfigVersion(PortalIp, PortalPort, ApiKey, ServiceType, Version, logger=None): + """ + Update config version + :param PortalIp: + :param PortalPort: + :param ServiceType: + :param ConfigFilePath: + :param logger: + :return: + """ + + Url = "/api/v1/Config/%s/%s" % (ServiceType, Version) + body = json.dumps(dict()) + Conn = RestApi(PortalIp, PortalPort, Url, authkey=ApiKey, params=body, logger=logger) + Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + + +@catch_exceptions() +def SetServiceConfig(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, ConfFile=None, logger=None): + # Get Current Service config + Ret, Conf = GetConfig(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger, ConfFile=ConfFile) + if Ret is False: + return Ret, Conf, None + + Url = "/api/v1/Config/%s" % ServiceType + body = Conf + Conn = RestApi(IfsPortalIp, IfsPortalPort, Url, authkey=ApiKey, params=body, logger=logger) + Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) + return Res, Errmsg, Data + + +@catch_exceptions() +def GetLatestServiceConfig(PortalIp, PortalPort, PortalApiKey, ServiceType, logger=None): + ServiceType = ServiceTypeConversion[ServiceType.lower()] + Res, Errmsg, Ret, Data = GetServiceConfigList(PortalIp, PortalPort, PortalApiKey, ServiceType, logger=logger) + if Res != ResOk: + print(Errmsg) + return None, Errmsg + else: + if Ret.Result == ResultSuccess: + LatestVersionId = 0 + LatestConfig = None + for Config in Data.Items: + if Config.Version > LatestVersionId: + LatestVersionId = Config.Version + LatestConfig = Config + return LatestConfig, '' + else: + return None, Ret.Message + +@catch_exceptions() +def RemoveServiceConfig(PortalIp, PortalPort, ApiKey, ServiceType, Version, logger=None): + """ + delete service config from config list + :param ip: + :param port: + :param Id: + :param logger: + :return:tuple(error code, error msg, ResponseHeader class) + """ + Url = '/api/v1/Config/%s/%s' % (ServiceType, Version) + Conn = RestApi(PortalIp, PortalPort, Url, authkey=ApiKey, logger=logger) + Res, Errmsg, Ret = Conn.delete() + return Res, Errmsg, Ret + + +def GetConfig(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger, ConfFile=None): + if ConfFile is None: + Conf = GetConfigFromUser(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger) + Conf = json.dumps(Conf) + Conf = Conf.replace('"', '\\"') + Conf = '"'+ Conf + '"' + return True, Conf + else: + if os.path.exists(ConfFile): + Ret, Conf = GetConf(ConfFile, FileType='non-ini', ReturnType='dict') + if Ret is True: + Conf = json.dumps(Conf) + Conf = Conf.replace('"', '\\"') + Conf = '"' + Conf + '"' + return True, Conf + else: + return Ret, Conf + else: + return False, '%s is not found' % ConfFile + + +def GetConfigFromUser(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger): + + ConfigInfo = GetServiceCurrentConfigInfo(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger) + + Conf = dict() + for info in ConfigInfo: + ConfKey = info['key'] + QuestionString = info['question'] + ValueType = info['type'] + DefaultValue = info['value'] + if 'valid_answer_list' in info: + ValueAnswerList = info['valid_answer_list'] + else: + ValueAnswerList = None + if 'Activate' in info: + if info['Activate'] is False: + Conf[ConfKey] = info['value'] + continue + ValueComment = '' + if 'value_command' in info: + ValueComment = info['value_command'] + + Conf[ConfKey]= get_input(QuestionString, ValueType, DefaultValue, ValidAnsList=ValueAnswerList, ValueComment=ValueComment) + return Conf + + +def GetServiceCurrentConfigInfo(PortalIp, PortalPort, PortalApiKey, ServiceType, logger): + # Get MqConfig + """ + Get Current Config if exists, or Default Config if not exists. + :param PortalIp: + :param PortalPort: + :param PortalApiKey: + :param ServiceType: + :param logger: + :return: + """ + + Res, Errmsg, Ret, Data = GetServiceConfig(PortalIp, PortalPort, PortalApiKey, ServiceType, logger=logger) + if Res != ResOk: + logger.error('fail to get config %s' % Errmsg) + else: + if (Ret.Result == ResultSuccess): + DsPServiceConf(Data) + else: + logger.error('fail to get config %s' % Ret.Message) + + ConfigInfo = None + if ServiceType == TypeServiceOSD: + if Data: + CurrentConf = json.loads(Data.Config) + for info in OsdDefaultConfigInfo: + try: + ConfKey = info['key'] + info['value'] = CurrentConf[ConfKey] + except Exception as err: + pass + ConfigInfo = OsdDefaultConfigInfo + + elif ServiceType == TypeServiceGW: + + if Data: + + CurrentConf = json.loads(Data.Config) + for info in S3DefaultConfigInfo: + try: + ConfKey = info['key'] + info['value'] = CurrentConf[ConfKey] + except Exception as err: + pass + else: + + # Get RaggitMq Config + Ret, MqConfig = GetBaseConfig(PortalIp, PortalPort, PortalApiKey, TypeServiceRabbitMq, logger) + if Ret is False: + return None + CurrentMqConfig = json.loads(MqConfig.Config) + + # Get MariaDB Config + Ret, MariaDBConfig = GetBaseConfig(PortalIp, PortalPort, PortalApiKey, TypeServiceMariaDB, logger) + if Ret is False: + return None + + CurrentMariaDBConfig = json.loads(MariaDBConfig.Config) + + UpdateGwBaseConfig(S3DefaultConfigInfo, CurrentMqConfig, CurrentMariaDBConfig) + + ConfigInfo = S3DefaultConfigInfo + + elif ServiceType == TypeServiceMongoDB: + if Data: + CurrentConf = json.loads(Data.Config) + for info in OsdDefaultConfigInfo: + try: + ConfKey = info['key'] + if ConfKey == 'objM.mq_host': + info['value'] = CurrentConf['Host'] + elif ConfKey == 'objM.mq_osd_exchange_name': + info['value'] = CurrentConf['ExchangeName'] + except Exception as err: + pass + ConfigInfo = MongoDbConfigInfo + + elif ServiceType in [TypeServiceLifecycle, TypeServiceReplication, TypeServiceLogManager]: + if Data: + CurrentConf = json.loads(Data.Config) + for info in BackendConfigInfo[ServiceType]: + try: + ConfKey = info['key'] + info['value'] = CurrentConf[ConfKey] + except Exception as err: + pass + ConfigInfo = BackendConfigInfo[ServiceType] + + + return ConfigInfo + +def GetBaseConfig(PortalIp, PortalPort, PortalApiKey, ConfigType, logger ): + """ + Get Base(Mariadb, RabbitMq) Config + :param PortalIp: + :param PortalPort: + :param PortalApiKey: + :param ConfigType: + :param logger: + :return: + """ + + Res, Errmsg, Ret, MqConfig = GetServiceConfig(PortalIp, PortalPort, PortalApiKey, ConfigType, logger=logger) + if Res != ResOk: + logger.error('fail to get %s config %s' % (ConfigType, Errmsg)) + else: + if (Ret.Result == ResultSuccess): + #DsPServiceConf(MqConfig) + return True, MqConfig + else: + logger.error('fail to get %s config %s' % (ConfigType, Ret.Message)) + return False, None + +def UpdateGwBaseConfig(GwConfig, MqConfig, MariaDBConfig): + """ + Merge GwConfig and MqConfig. + :param GwConfig: + :param MqConfig: + :return: + """ + MqKeyMap = {"objM.mq_host": "Host", "objM.mq_exchange_name": "ExchangeName"} + MariaDBKeyMap = {"gw.db_host": "Host", "gw.db_name": "Name", "gw.db_port": "Port", "gw.db_user": "User", + "gw.db_password": "Password", "objM.db_host": "Host", "objM.db_name": "Name", "objM.db_port": "Port", + "objM.db_user=root": "User", "objM.db_password": "Password"} + + + for info in GwConfig: + try: + ConfKey = info['key'] + if ConfKey in MqKeyMap: + BaseKey = MqKeyMap[ConfKey] + info['value'] = MqConfig[BaseKey] + elif ConfKey in MariaDBKeyMap: + BaseKey = MariaDBKeyMap[ConfKey] + info['value'] = MariaDBConfig[BaseKey] + + except Exception as err: + print('fail to update ksanGW Mq & MariaDB Config %s' % str(err)) + +def isKsanServiceIdFileExists(ServiceType): + """ + check if systemd service file exists or not in local server or docker + :param ServiceType: + :return: if File exists: True, else False + """ + if ServiceType in ServiceTypeServiceHiddenPathMap: + isDockerService = False + + # check if the service is docker support or not from system service file + if ServiceType in ServiceTypeSystemdServiceMap: + SystemdServiceUnitPath = ServiceTypeSystemdServiceMap[ServiceType] + try: + with open('/etc/systemd/system/'+ SystemdServiceUnitPath, 'r') as f: + SystemdServiceContens = f.read() + if 'Requires=docker.service' in SystemdServiceContens: + isDockerService = True + except Exception as err: + print('fail to read system service file %s %s.' % (SystemdServiceUnitPath, str(err))) + return False, 'fail to read system service file %s %s.' % (SystemdServiceUnitPath, str(err)) + + + ServiceHiddenPath = ServiceTypeServiceHiddenPathMap[ServiceType] + + if isDockerService is False: + if os.path.exists(ServiceHiddenPath): + return True, '' + else: + return False, '' + else: + # copy from docker to local and check if serviceid file exists or not + CopiedServiceIdFile = '/tmp/.TmpServiceId%d' % int(time.time()) + ServiceContainerName = ServiceTypeDockerServiceContainerNameMap[ServiceType] + DockerGetServiceFileCmd = 'docker cp %s:%s %s' % (ServiceContainerName, ServiceHiddenPath, CopiedServiceIdFile) + shcall(DockerGetServiceFileCmd) + if os.path.exists(CopiedServiceIdFile): + return True, '' + else: + return False, '' + else: + return True, '' + + +def SaveKsanServiceIdFile(ServiceType, ServiceId): + if ServiceType in ServiceTypeServiceHiddenPathMap: + ServiceHiddenPath = ServiceTypeServiceHiddenPathMap[ServiceType] + isDockerService = False + + if ServiceType in ServiceTypeSystemdServiceMap: + SystemdServiceUnitPath = ServiceTypeSystemdServiceMap[ServiceType] + try: + with open('/etc/systemd/system/'+ SystemdServiceUnitPath, 'r') as f: + SystemdServiceContens = f.read() + if 'Requires=docker.service' in SystemdServiceContens: + isDockerService = True + except Exception as err: + return False, 'fail to read systemd service file %s %s' % (SystemdServiceUnitPath, str(err)) + + try: + with open(ServiceHiddenPath, 'w') as f: + f.write(ServiceId) + f.flush() + except Exception as err: + return False, 'fail to create ServiceId file %s %s' % (ServiceHiddenPath, str(err)) + + if isDockerService is True: + ServiceContainerName = ServiceTypeDockerServiceContainerNameMap[ServiceType] + try: + SystemdServiceUnitPath = ServiceTypeSystemdServiceMap[ServiceType] + ServiceStopCmd = 'systemctl stop %s' % SystemdServiceUnitPath + shcall(ServiceStopCmd) + DockerCopyCmd = 'docker cp %s %s:%s' % (ServiceHiddenPath, ServiceContainerName, ServiceHiddenPath) + shcall(DockerCopyCmd) + os.unlink(ServiceHiddenPath) + return True, '' + except Exception as err: + return False, 'fail to create ServiceId file %s:%s %s' % (ServiceContainerName, ServiceHiddenPath, str(err)) + else: + return True, '' + else: + return False, 'service type %s is invalid' % ServiceType + +@catch_exceptions() +def MqServiceHandler(MonConf, RoutingKey, Body, Response, ServerId, ServiceList, LocalIpList, GlobalFlag, logger): + """ + Message Queue Service Handler + """ + logger.debug("%s %s %s" % (str(RoutingKey), str(Body), str(Response))) + ResponseReturn = MqReturn(ResultSuccess) + Body = Body.decode('utf-8') + Body = json.loads(Body) + body = DictToObject(Body) + + # Service Control Handle + if RoutKeyServiceControlFinder.search(RoutingKey): + logging.log(logging.INFO, "%s" % str(Body)) + #if body.Id not in ServiceList['IdList']: + # LoadServiceList(MonConf, ServiceList, LocalIpList, logger) + + Response.IsProcessed = True + #if body.Id not in ServiceList['Details']: + # logger.error('Invalid Service Id %s' % body.Id) + # return ResponseReturn + ServiceType = body.ServiceType + #if ServiceType == TypeHaproxy: + # Ret = Hap.StartStop(Body) + #elif ServiceType == TypeS3: + # Ret = S3.StartStop(Body) + service = ServiceUnit(logger, ServiceType) + if body.Control == START: + Ret, ErrMsg = service.Start() + elif body.Control == STOP: + Ret, ErrMsg = service.Stop() + elif body.Control == RESTART: + Ret, ErrMsg = service.Restart() + elif ServiceType.lower() == TypeServiceAgent.lower(): + Ret = True + ErrMsg = '' + else: + Ret = False + ErrMsg = 'Invalid Serivce Type' + + if Ret is False: + ResponseReturn = MqReturn(Ret, Code=1, Messages=ErrMsg) + logger.error(ResponseReturn) + else: + logging.log(logging.INFO, ResponseReturn) + return ResponseReturn + # Serivce Config Handle + elif ".config." in RoutingKey: + if body.Id in ServiceList['IdList']: + Response.IsProcessed = True + logger.debug(ResponseReturn) + return ResponseReturn + elif RoutingKey.endswith((".added", ".updated", ".removed")): + ServiceType = body.ServiceType + GlobalFlag['ServiceUpdated'] = Updated + + if RoutingKey.endswith(".added"): + ret, errlog = isKsanServiceIdFileExists(ServiceType) + if ret is False: + logger.debug('ServiceId file is not found. %s %s' % (body.Id, errlog)) + + ret = IsValidServiceIdOfLocalServer(MonConf, logger, body.Id) + if ret is True: + ret, errlog = SaveKsanServiceIdFile(ServiceType, body.Id) + if ret is False: + logger.error(errlog) + + Response.IsProcessed = True + logger.debug(ResponseReturn) + return ResponseReturn + else: + Response.IsProcessed = True + return ResponseReturn + diff --git a/core/common/service/service_manage.py b/core/common/service/service_manage.py index 5b4811ba..eedc1161 100644 --- a/core/common/service/service_manage.py +++ b/core/common/service/service_manage.py @@ -16,1234 +16,234 @@ import sys, os import socket import time -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from server.server_manage import * -from const.http import ResponseHeaderModule, ResponseItemsHeaderModule -from const.service import AddServiceInfoObject, ServiceDetailModule, \ - ServiceItemsDetailModule, ServiceConfigModule, UpdateServiceInfoObject, UpdateServiceUsageObject, \ - AddServiceGroupItems, ServiceGroupItemsModule -from common.init import get_input, GetConf, GetAgentConfig -from common.utils import DisplayState -from common.shcommand import shcall -from const.common import ServiceTypeServiceHiddenPathMap, ServiceTypeSystemdServiceMap, ServiceTypeDockerServiceContainerNameMap -import logging - -ConfigMaxWidthLen = 100 -ConfigMaxValueLen = 100 - 13 -ConfigCompartLine = '%s' % ('=' * ConfigMaxWidthLen) -ConfigTitleLine = '%s' % ('-' * ConfigMaxWidthLen) - - -OsdDefaultPoolsize = 650 -OsdDefaultPort = 8000 -OsdDefaultEcScheduleMinute = 30000 -OsdDefaultEcApplyMinute = 3000000 -OsdDefaultEcFileSize = 100000 -OsdDefaultCacheDisk = '' -OsdDefaultCacheScheduleMinute = 2 -OsdDefaultCacheFileSize = 1024 -OsdDefaultCacheLimitMinute = 5 -OsdDefaultTrashScheduleMinute = 5 -OsdDefaultPerformanceMode = 'NO_OPTION' - -OsdDefaultConfigInfo = [{'key': 'osd.pool_size', 'value': OsdDefaultPoolsize, 'type': int, 'question': 'Insert connection pool size'}, - {'key': 'osd.port', 'value': OsdDefaultPort, 'type': int, 'question': 'Insert ksanOSD port'}, - {'key': 'osd.ec_check_interval', 'value': OsdDefaultEcScheduleMinute, 'type': int, 'question': 'Insert EC check interval(ms)'}, - {'key': 'osd.ec_wait_time', 'value': OsdDefaultEcApplyMinute, 'type': int, 'question': 'Insert EC wait time(ms)'}, - {'key': 'osd.ec_min_size', 'value': OsdDefaultEcFileSize, 'type': int, 'question': 'Insert EC file size'}, - {'key': 'osd.cache_diskpath', 'value': OsdDefaultCacheDisk, 'type': str, 'question': 'Insert cache disk path'}, - {'key': 'osd.cache_check_interval', 'value': OsdDefaultCacheScheduleMinute, 'type': int, 'question': 'Insert cache check interval(ms)'}, - {'key': 'osd.cache_expire', 'value': OsdDefaultCacheLimitMinute, 'type': int, 'question': 'Insert cache expire(ms)'}, - {'key': 'osd.trash_check_interval', 'value': OsdDefaultTrashScheduleMinute, 'type': int, 'question': 'Insert trash check interval(ms)'}] - - -S3DefaultDbRepository = TypeServiceMariaDB -S3DefaultDbHost = '127.0.0.1' -S3DefaultDatabase = 'ksan' -S3DefaultDbPort = 3306 -S3DefaultDbUser = 'root' -S3DefaultDbPassword = 'YOUR_DB_PASSWORD' -S3DefaultDbPoolSize = 20 -S3DefaultGwAuthrization = 'AWS_V2_OR_V4' -S3DefaultGwEndpoint = 'http://0.0.0.0:8080' -S3DefaultGwSecureEndpoint = 'https://0.0.0.0:8443' -S3DefaultGwKeyStorePath = '/usr/local/ksan/ssl/pspace.jks' -S3DefaultGwtKeyStorePassword = 'YOUR_JKS_PASSWORD' -S3DefaultGwMaxFileSize = 3221225472 -S3DefaultGwMaxListSize = 200000 -S3DefaultGwMaxTimeSkew = 9000 -S3DefaultGwReplication = 2 -S3DefaultGwOsdPort = 8000 -S3DefaultGwJettyMaxThreads = 1000 -S3DefaultGwJettyMaxIdleTime = 600000 -S3DefaultGwOsdClientCount = 100 -S3DefaultGwObjmanagerCount = 100 -S3DefaultGwPerformanceMode = 'NO_OPTION' -S3DefaultCacheDisk = '' -S3DefaultCacheFileSize = 1024 - -ObjManagerDefaultDbRepository = TypeServiceMariaDB -ObjManagerDefaultDbHost = '127.0.0.1' -ObjManagerDefaultDatabase = 'ksan' -ObjManagerDefaultDbPort = 3306 -ObjManagerDefaultDbUser = 'root' -ObjManagerDefaultDbPassword = 'YOUR_DB_PASSWORD' -ObjManagerDefaultMqHost = '127.0.0.1' -ObjManagerDefaultMqName = 'disk' -ObjManagerDefaultMqExchangeName = 'ksan.system' -ObjManagerDefaultMqOsdExchangeName = 'OSDExchange' - -S3DefaultConfigInfo = [ - {'key': 'gw.db_repository', 'value': S3DefaultDbRepository, 'type': str, 'question': 'Insert ksanGW DB repository', 'valid_answer_list': [TypeServiceMariaDB, TypeServiceMongoDB]}, - {'key': 'gw.db_host', 'value': S3DefaultDbHost, 'type': str, 'question': 'Insert ksanGW DB host'}, - {'key': 'gw.db_name', 'value': S3DefaultDatabase, 'type': str, 'question': 'Insert ksanGW DB name'}, - {'key': 'gw.db_port', 'value': S3DefaultDbPort, 'type': int, 'question': 'Insert ksanGW DB port'}, - {'key': 'gw.db_user', 'value': S3DefaultDbUser, 'type': str, 'question': 'Insert ksanGW DB user'}, - {'key': 'gw.db_password', 'value': S3DefaultDbPassword, 'type': str, 'question': 'Insert ksanGW DB password'}, - {'key': 'gw.db_pool_size', 'value': S3DefaultDbPoolSize, 'type': int, 'question': 'Insert ksanGW DB pool size'}, - {'key': 'gw.authorization', 'value': S3DefaultGwAuthrization, 'type': str, 'question': 'Insert authorization'}, - {'key': 'gw.endpoint', 'value': S3DefaultGwEndpoint, 'type': 'url', 'question': 'Insert endpoint url'}, - {'key': 'gw.secure_endpoint', 'value': S3DefaultGwSecureEndpoint, 'type': 'url', - 'question': 'Insert secure endpoint url'}, - {'key': 'gw.keystore_path', 'value': S3DefaultGwKeyStorePath, 'type': str, 'question': 'Insert keystore path'}, - {'key': 'gw.keystore_password', 'value': S3DefaultGwtKeyStorePassword, 'type': str, - 'question': 'Insert keystore password'}, - {'key': 'gw.max_file_size', 'value': S3DefaultGwMaxFileSize, 'type': int, 'question': 'Insert max file size'}, - {'key': 'gw.max_list_size', 'value': S3DefaultGwMaxListSize, 'type': int, 'question': 'Insert max sist size'}, - {'key': 'gw.max_timeskew', 'value': S3DefaultGwMaxTimeSkew, 'type': int, 'question': 'Insert max time skew'}, - {'key': 'gw.osd_port', 'value': S3DefaultGwOsdPort, 'type': int, 'question': 'Insert ksanOSD port'}, - {'key': 'gw.jetty_max_threads', 'value': S3DefaultGwJettyMaxThreads, 'type': int, - 'question': 'Insert Jetty max thread count'}, - {'key': 'gw.jetty_max_idle_timeout', 'value': S3DefaultGwJettyMaxIdleTime, 'type': int, - 'question': 'Insert Jetty max idle time'}, - {'key': 'gw.osd_client_count', 'value': S3DefaultGwOsdClientCount, 'type': int, - 'question': 'Insert ksanOSD client count'}, - {'key': 'gw.objmanager_count', 'value': S3DefaultGwObjmanagerCount, 'type': int, - 'question': 'Insert ksanOBJMANAGER count'}, - {'key': 'gw.performance_mode', 'value': S3DefaultGwPerformanceMode, 'type': str, - 'question': 'Insert performance mode'}, - {'key': 'gw.cache_diskpath', 'value': S3DefaultCacheDisk, 'type': str, 'question': 'Insert cache disk path'}, - {'key': 'gw.cache_file_size', 'value': S3DefaultCacheFileSize, 'type': int, 'question': 'Insert cache file size'}, - {'key': 'objM.db_repository', 'value': ObjManagerDefaultDbRepository, 'type': str, - 'question': 'Insert object DB repository'}, - {'key': 'objM.db_host', 'value': ObjManagerDefaultDbHost, 'type': str, 'question': 'Insert object DB host', 'valid_answer_list': [TypeServiceMariaDB, TypeServiceMongoDB]}, - {'key': 'objM.db_name', 'value': ObjManagerDefaultDatabase, 'type': str, 'question': 'Insert object DB name'}, - {'key': 'objM.db_port', 'value': ObjManagerDefaultDbPort, 'type': int, 'question': 'Insert object DB port'}, - {'key': 'objM.db_user', 'value': ObjManagerDefaultDbUser, 'type': str, 'question': 'Insert object DB user'}, - {'key': 'objM.db_password', 'value': ObjManagerDefaultDbPassword, 'type': str, 'question': 'Insert object DB password'} - #{'key': 'objM.mq_host', 'value': ObjManagerDefaultMqHost, 'type': str, 'question': 'Insert MQ host', 'Activate': False}, - #{'key': 'objM.mq_queue_name', 'value': ObjManagerDefaultMqName, 'type': str, 'question': 'Insert MQ name', 'Activate': False}, - #{'key': 'objM.mq_exchange_name', 'value': ObjManagerDefaultMqExchangeName, 'type': str, 'question': 'Insert MQ exchange name', 'Activate': False}, - #{'key': 'objM.mq_osd_exchange_name', 'value': ObjManagerDefaultMqOsdExchangeName, 'type': str, 'question': 'Insert MQ ksanOSD exchange name', 'Activate': False} - #{'key': 'objM.mq.host', 'value': ObjManagerDefaultMqHost, 'type': 'ip', 'question': 'Insert MQ Host'}, - #{'key': 'objM.mq.diskpool.queuename', 'value': ObjManagerDefaultMqDiskpoolQueueName, 'type': str, - # 'question': 'Insert MQ Disk pool Queue Name'}, - #{'key': 'objM.mq.diskpool.exchangename', 'value': ObjManagerDefaultMqDiskpoolExchangeName, 'type': str, - # 'question': 'Insert MQ Disk pool Exchange Name'}, - #{'key': 'objM.mq.osd.exchangename=OSDExchange', 'value': ObjManagerDefaultMqOsdExchangeName, 'type': str, - # 'question': 'Insert MQ ksanOSD Exchange Name'} -] - -MongoDbDefaultShard1Port = 20001 -MongoDbDefaultShard2Port = 20002 -MongoDbDefaultConfigServerPort = 50000 -MongoDbDefaultDbPort = 27017 -MongoDbDefaultHomeDir = '/var/lib/mongo' -MongoDbDefaultPrimaryHost = socket.gethostname() - -MongoDbConfigInfo = [ - {'key': 'Shard1Port', 'value': MongoDbDefaultShard1Port, 'type': int, 'question': 'Insert MongoDB shard1 port'}, - {'key': 'Shard2Port', 'value': MongoDbDefaultShard2Port, 'type': int, 'question': 'Insert MongoDB shard2 port'}, - {'key': 'ConfigServerPort', 'value': MongoDbDefaultConfigServerPort, 'type': int, 'question': 'Insert MongoDB config server port'}, - {'key': 'MongoDbPort', 'value': MongoDbDefaultDbPort, 'type': int, - 'question': 'Insert MongoDB DB port'}, - {'key': 'HomeDir', 'value': MongoDbDefaultHomeDir, 'type': str, - 'question': 'Insert MongoDB home directory'}, - {'key': 'PrimaryHostName', 'value': MongoDbDefaultPrimaryHost, 'type': str, - 'question': 'Insert MongoDB primary cluster node hostname'} -] - -lifecycleDefaultDbRepository = TypeServiceMariaDB -lifecycleDefaultDbHost = '127.0.0.1' -lifecycleDefaultDbPort = 3306 -lifecycleDefaultDbName = 'ksan' -lifecycleDefaultDbUser = 'root' -lifecycleDefaultDbPassword = 'YOUR_DB_PASSWORD' -lifecycleDefaultRegion = 'kr-ksan-1' -lifecycleDefaultRuntime = '01:00' -lifecycleDefaultCheckInterval = 5000 - -LifecycleDefaultConfigInfo = [{'key': 'objM.db_repository', 'value': lifecycleDefaultDbRepository, 'type': str, 'question': 'Insert ksanGW DB repository'}, - {'key': 'objM.db_host', 'value': lifecycleDefaultDbHost, 'type': str, 'question': 'Insert ksanGW DB host'}, - {'key': 'objM.db_port', 'value': lifecycleDefaultDbPort, 'type': int, 'question': 'Insert ksanGW DB port'}, - {'key': 'objM.db_name', 'value': lifecycleDefaultDbName, 'type': str, 'question': 'Insert ksanGW DB name'}, - {'key': 'objM.db_user', 'value': lifecycleDefaultDbUser, 'type': str, 'question': 'Insert ksanGW DB user'}, - {'key': 'objM.db_password', 'value': lifecycleDefaultDbPassword, 'type': str, 'question': 'Insert ksanGW DB password'}, - {'key': 'ksan.region', 'value': lifecycleDefaultRegion, 'type': str, 'question': 'Insert region'}, - {'key': 'lifecycle.schedule', 'value': lifecycleDefaultRuntime, 'type': str, 'question': 'Insert runtime'}, - {'key': 'lifecycle.check_interval', 'value': lifecycleDefaultCheckInterval, 'type': int, 'question': 'Insert check interval'}] - -@catch_exceptions() -def AddService(MgsIp, Port, ApiKey, ServiceName, ServiceType, ServerId=None, VlanIds=[], ServerName=None, GroupId='', Description='', logger=None): - #VlanIds = GetVlanIdListFromServerDetail(MgsIp, Port, ApiKey, ServerId=ServerId, ServerName=ServerName, logger=logger) - #if VlanIds is None: - # return ResEtcErrorCode, ResFailToGetVlainId, None - if ServerId is not None: - TargetServer = ServerId - elif ServerName is not None: - TargetServer = ServerName - else: - return ResInvalidCode, ResInvalidMsg + 'ServerName is required', None - - - Service = AddServiceInfoObject() - Service.Set(ServiceName, TargetServer, ServiceType, GroupId, VlanIds, State='Offline', Description=Description, HaAction='Initializing') - if ServiceType is None: - return ResInvalidCode, ResInvalidMsg + 'ServiceType is required', None - - Url = "/api/v1/Services" - body = jsonpickle.encode(Service, make_refs=False) - Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=body, logger=logger) - Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - -def RegisterService(conf, ServiceType, logger): - """ - - :param Conf: - :param ServiceType: - :param logger: - :return: - """ - out, err = shcall('hostname') - ServiceName = out[:-1] + '_%s' % ServiceType - Res, Errmsg, Ret = AddService(conf.PortalHost, conf.PortalPort, conf.PortalApiKey, ServiceName, ServiceType, - conf.ServerId, logger=logger) - if Res == ResOk: - if Ret.Result != ResultSuccess: - if Ret.Result != CodeDuplicated: - logger.error("%s %s" % (str(Ret.Result), str(Ret.Message))) - else: - logging.log(logging.INFO, "%s is registered. %s %s" % (ServiceType, str(Ret.Result), str(Ret.Message))) - else: - logger.error("%s %s" % (str(Ret.Result), str(Ret.Message))) - -def KsanServiceRegister(conf, ServiceType, logger): - Retry = 0 - while True: - Retry += 1 - Res, Errmsg, Ret, Data = GetServerInfo(conf.PortalHost, conf.PortalPort,conf.PortalApiKey , ServerId=conf.ServerId, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - NetworkInterfaces = Data.NetworkInterfaces - if len(NetworkInterfaces) > 0: - isNetworkAdded = True - break +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from const.common import * +import xml.etree.ElementTree as ET +from const.service import UpdateServicesStateObject, SystemdServicePidFinder +import mqmanage.mq +from portal_api.apis import * + + + +class Process: + def __init__(self,ServiceType): + self.ServiceType = ServiceType + self.SystemdServiceName = ServiceTypeSystemdServiceMap[ServiceType] if ServiceType in ServiceTypeSystemdServiceMap else None + self.Pid = None + self.CpuUsage = 0 + self.MemoryUsed = 0 + self.ThreadCount = 0 + self.State = 'Offline' + self.GetPidWithServiceType() + self.GetState() + self.GetUsage() + + def GetPidWithServiceType(self): + if self.SystemdServiceName is not None: + SystemdCmd = 'systemctl status %s' % self.SystemdServiceName + out, err = shcall(SystemdCmd) + Pid = SystemdServicePidFinder.search(out) + if not Pid: + self.Pid = None else: - logger.error('fail to get network interface %s' % Ret.Message) + self.Pid = int(Pid.groups()[0]) else: - logger.error('fail to get network interface %s' % Errmsg) - if Retry > ServiceContolRetryCount: - isNetworkAdded = False - break - time.sleep(IntervalMiddle) - - if isNetworkAdded is True: - RegisterService(conf, ServiceType, logger) - -@catch_exceptions() -def DeleteService(ip, port, ApiKey, ServiceId=None, ServiceName=None, logger=None): - """ - delete serviceinfo from server pool - :param ip: - :param port: - :param Id: - :param logger: - :return:tuple(error code, error msg, ResponseHeader class) - """ - if ServiceId is not None: - TargetServcie = ServiceId - elif ServiceName is not None: - TargetServcie = ServiceName - else: - return ResInvalidCode, ResInvalidMsg, None - - Url = '/api/v1/Services/%s' % TargetServcie - Conn = RestApi(ip, port, Url, authkey=ApiKey, logger=logger) - Res, Errmsg, Ret = Conn.delete() - return Res, Errmsg, Ret - - -@catch_exceptions() -def GetServiceInfo(MgsIp, Port, ApiKey, ServiceId=None, ServiceName=None, logger=None): - """ - get service info all or specific service info with Id - :param ip: - :param port: - :param disp: - :return: - """ - if ServiceId is not None: - TargetService = ServiceId - elif ServiceName is not None: - TargetService = ServiceName - else: - TargetService = None - - if TargetService is not None: - Url = "/api/v1/Services/%s" % TargetService - ItemsHeader = False - ReturnType = ServiceDetailModule - else: - Url = "/api/v1/Services/" - ItemsHeader = True - ReturnType = ServiceItemsDetailModule - Params = dict() - Params['countPerPage'] = 100 - Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=Params, logger=logger) - Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) - if Res == ResOk: - AllServiceInfo = list() - if Ret.Result == ResultSuccess: - if TargetService is not None: - return Res, Errmsg, Ret, Ret.Data - else: - return Res, Errmsg, Ret, Ret.Data.Items - return Res, Errmsg, Ret, None - else: - return Res, Errmsg, None, None - -@catch_exceptions() -def GetServiceMongoDBConfig(MgsIp, Port, ApiKey, logger=None): - """ - get service info all or specific service info with Id - :param ip: - :param port: - :param disp: - :return: - """ - Url = "/api/v1/Config/MongoDB" - ItemsHeader = False - ReturnType = ServiceConfigModule - - Params = dict() - Params['countPerPage'] = 100 - Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=Params, logger=logger) - Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) - if Res == ResOk: - AllServiceInfo = list() - if Ret.Result == ResultSuccess: - return Res, Errmsg, Ret, Ret.Data - return Res, Errmsg, Ret, None - else: - return Res, Errmsg, None, None - - -@catch_exceptions() -def UpdateServiceInfo(MgsIp, Port, ApiKey, ServiceId=None, ServiceName=None, GroupId=None, Description=None, ServiceType=None, - HaAction=None, State=None, logger=None): - Res, Errmsg, Ret, Service = GetServiceInfo(MgsIp, Port, ApiKey, ServiceId=ServiceId, ServiceName=ServiceName, logger=logger) - if Res != ResOk: - return Res, Errmsg, None - else: - if Ret.Result != ResultSuccess: - return Res, Errmsg, Ret - - VlanIds = GetVlanIdListFromVlanList(Service.Vlans) - NewService = UpdateServiceInfoObject() - NewService.Set(Service.Name, Service.ServiceType, Service.GroupId, VlanIds, Service.State, - Service.Description, Service.HaAction) - - if ServiceName is not None: - NewService.Name = ServiceName - if GroupId is not None: - NewService.GroupId = GroupId - if Description is not None: - NewService.Description = Description - if ServiceType is not None: - NewService.ServiceType = ServiceType - if HaAction is not None: - NewService.HaAction = HaAction - if VlanIds is not None: - NewService.VlanIds = VlanIds - if State is not None: - NewService.State = State - - if ServiceId is not None: - TargetServcie = ServiceId - elif ServiceName is not None: - TargetServcie = ServiceName - else: - return ResInvalidCode, ResInvalidMsg, None - - Url = "/api/v1/Services/%s" % TargetServcie - body = jsonpickle.encode(NewService, make_refs=False) - Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=body, logger=logger) - Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - - - -@catch_exceptions() -def UpdateServiceState(MgsIp, Port, ApiKey, ServiceId, State, logger=None): - - Url = "/api/v1/Services/%s/State/%s" % (ServiceId, State) - body = dict() - Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=body, logger=logger) - Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - - -@catch_exceptions() -def UpdateServiceUsage(MgsIp, Port, ApiKey, ServiceId, logger=None): - Usage = UpdateServiceUsageObject() - Usage.Set(ServiceId, 48, 55555, 88) - Url = "/api/v1/Services/Usage" - body = jsonpickle.encode(Usage, make_refs=False) - Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey, params=body, logger=logger) - Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data + if self.ServiceType == TypeServiceAgent: + Ret, Pid = IsDaemonRunning(KsanAgentPidFile, CmdLine=KsanAgentName) + if Ret is True: + self.Pid = int(Pid) + else: + self.Pid = None + def GetUsage(self): + try: + p = psutil.Process(pid=self.Pid) + self.CpuUsage = p.cpu_percent(interval=1) + m = p.memory_full_info() + #self.MemoryUsed = m.rss + m.vms + m.shared + m.data + self.MemoryUsed = m.rss + self.ThreadCount = p.num_threads() + except Exception as err: + self.CpuUsage = 0 + self.MemoryUsed = 0 + self.ThreadCount = 0 -@catch_exceptions() -def ControlService(MgsIp, Port, ApiKey, Control, ServiceId=None, ServiceName=None, logger=None): - Res, Errmsg, Ret, Data = GetServiceInfo(MgsIp, Port, ApiKey, ServiceId=ServiceId, ServiceName=ServiceName, logger=logger) - if Res != ResOk: - return Res, Errmsg, None - else: - if Ret.Result != ResultSuccess: - return Res, Errmsg, Ret - if ServiceId is not None: - TargetServcie = ServiceId - elif ServiceName is not None: - TargetServcie = ServiceName - else: - return ResInvalidCode, ResInvalidMsg, None + def GetState(self): + if self.Pid is not None: + self.State = ONLINE + else: + self.State = OFFLINE - Url = "/api/v1/Services/%s/%s" % (TargetServcie, Control) - body = dict() - Conn = RestApi(MgsIp, Port, Url, authkey=ApiKey,params=body, logger=logger) - Res, Errmsg, Data = Conn.post() - return Res, Errmsg, Data @catch_exceptions() -def ShowServiceInfo(Data, Detail=False): +def ServiceMonitoring(Conf, GlobalFlag, logger): - #if Detail: - # ServiceTitle = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ('Name'.center(10), 'Id'.center(38), - # 'GruopId'.center(38), 'State'.center(10), 'Type'.center(10), - # 'CpuUsage'.center(20), 'MemoryUsed'.center(20),'ThreadCount'.center(10), 'ServerId'.center(38)) - # ServiceTitleLine = '%s' % ('=' * 212) - #else: - #ServiceTitle ="|%s|%s|%s|%s|%s|" % ('Name'.center(20), 'State'.center(10), 'Id'.center(38), 'Type'.center(10), 'GroupId'.center(38)) - ServiceTitle ="|%s|%s|%s|%s|" % ('Name'.center(20), 'State'.center(10), 'Id'.center(38), 'Type'.center(20)) - ServiceTitleLine = '%s' % ('=' * 93) - - print(ServiceTitleLine) - print(ServiceTitle) - print(ServiceTitleLine) - - for idx, svc in enumerate(Data): - #if Detail: - # _svc ="|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % \ - # (svc.Name.center(10), - # svc.Id.center(38), str(svc.GroupId).center(38), - # svc.State.center(10), str(svc.ServiceType).center(10), str(svc.CpuUsage).center(20) , - # str(svc.MemoryUsed).center(20), str(int(svc.ThreadCount)).center(10), svc.ServerId.center(38)) - # ServiceDataLine = '%s' % ('-' * 209) - #else: - _svc = "|%s|%s|%s|%s|" % \ - (svc.Name.center(20), svc.State.center(10), svc.Id.center(38), svc.ServiceType.center(20)) - #_svc = "|%s|%s|%s|%s|%s|" % \ - # (svc.Name.center(20), svc.State.center(10), svc.Id.center(38), svc.ServiceType.center(10), str(svc.GroupId).center(38)) - ServiceDataLine = '%s' % ('-' * 93) - print(_svc) - print(ServiceDataLine) - -def TmpGetServiceInfoList(Ip, Port): - Res, Errmsg, Ret = GetServiceInfo(Ip, Port) - if Res == ResOk: - if Ret == ResultSuccess: - return Res, Errmsg, Ret.Data - else: - return Ret.Result, Ret.Message, None - return Res, Errmsg, None - - -@catch_exceptions() -def ShowServiceInfoWithServerInfo(ServerList, Detail=False, Ip=None, Port=None, SysinfoDisp=False): - #Ret, Errmsg , Data = TmpGetServiceInfoList(Ip, Port) - #if Ret != ResOk: - # print('Fail to get Service Info %s' % Errmsg) - # return + Conf = WaitAgentConfComplete(inspect.stack()[1][3], logger, CheckNetworkDevice=True, CheckNetworkId=True) - if SysinfoDisp is True: - ServiceTitleLine = '%s' % ('=' * 105) - ServiceTitle ="|%s|%s|%s|%s|%s|" % ('Service'.center(25), 'Server'.center(15),'Status'.center(17), 'Type'.center(20), ' ' * 22) - ServiceDataLine = '%s' % ('-' * 105) - else: - ServiceTitleLine = '%s' % ('=' * 245) - ServiceTitle = "|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % ( - 'ServerName'.center(20), 'ServiceName'.center(20), 'ServiceId'.center(38), - 'ServiceGruopId'.center(38), 'State'.center(10), 'Type'.center(20), - 'CpuUsage'.center(20), 'MemoryUsed'.center(20), 'Thread Cnt'.center(10), 'ServerId'.center(38)) - ServiceDataLine = '%s' % ('-' * 245) - print(ServiceTitleLine) - print(ServiceTitle) - print(ServiceTitleLine) + conf = GetAgentConfig(Conf) - for Svr in ServerList: - if len(Svr.Services) > 0: - for Svc in Svr.Services: - if SysinfoDisp is True: - _Svc = "|%s|%s|%s|%s|%s|" % (str(Svc.Name).center(25), Svr.Name.center(15), Svc.State.center(17), str(Svc.ServiceType).center(20), ' ' * 22) - else: - _Svc ="|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|" % \ - (Svr.Name.center(20), str(Svc.Name).center(20), - Svc.Id.center(38), str(Svc.GroupId).center(38), - DisplayState(Svc.State).center(10), str(Svc.ServiceType).center(20), str(Svc.CpuUsage).center(20) , - str(Svc.MemoryUsed).center(20), str(int(Svc.ThreadCount)).center(10), Svr.Id.center(38)) - print(_Svc) - print(ServiceDataLine) - else: - print('No service data') - print(ServiceDataLine) + #KsanServiceRegister(conf, TypeServiceAgent, logger) + MqServiceUsage = mqmanage.mq.Mq(conf.MQHost, int(conf.MQPort), MqVirtualHost, conf.MQUser, conf.MQPassword, + RoutKeyServiceUsage, ExchangeName, QueueName='', logger=logger) + MqServiceState = mqmanage.mq.Mq(conf.MQHost, int(conf.MQPort), MqVirtualHost, conf.MQUser, conf.MQPassword, + RoutKeyServiceState, ExchangeName, QueueName='', logger=logger) -@catch_exceptions() -def GetVlanIdListFromServerDetail(MgsIp, PortarPort, ApiKey, ServerId=None, ServerName=None, logger=None): - retry = 0 - VlanIds = list() + ServiceMonitorInterval = int(conf.ServiceMonitorInterval)/1000 + # local service list + LocalServices = list() # [{ 'Id': ServiceId, 'Type': 'Osd', 'ProcessObject':Object, 'IsEnable': True, 'GroupId': GroupId, 'Status': 'Online'}] while True: - retry += 1 - Res, Errmsg, Ret, Data = GetServerInfo(MgsIp, PortarPort, ApiKey, ServerId=ServerId, Name=ServerName, logger=logger) + Res, Errmgs, Ret, ServerDetail = GetServerInfo(conf.PortalHost, conf.PortalPort, conf.PortalApiKey, conf.ServerId, logger=logger) if Res == ResOk: if Ret.Result == ResultSuccess: - for net in Data.NetworkInterfaces: - for vlan in net.NetworkInterfaceVlans: - VlanIds.append(vlan.Id) - break - else: - if retry > 3: - break - else: - time.sleep(1) - return VlanIds - - -@catch_exceptions() -def GetVlanIdListFromVlanList(VlanList): - VlanIds = list() - for Vlan in VlanList: - VlanIds.append(Vlan.Id) - return VlanIds - - -@catch_exceptions() -def AddServiceGroup(Ip, Port, Name, ServiceType, ServiceIpAddress=None, ServiceIds=[], Description='', logger=None): - - Group = AddServiceGroupItems() - Group.Set(Name, ServiceType, ServiceIpAddress=ServiceIpAddress, ServiceIds=ServiceIds) - - Url = "/api/v1/ServiceGroups" - body = jsonpickle.encode(Group, make_refs=False) - Conn = RestApi(Ip, Port, Url, params=body, logger=logger) - Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - - -@catch_exceptions() -def RemoveServiceGroup(ip, port, ServiceId, logger=None): - """ - delete serviceinfo from server pool - :param ip: - :param port: - :param Id: - :param logger: - :return:tuple(error code, error msg, ResponseHeader class) - """ - Url = '/api/v1/ServiceGroups/%s' % ServiceId - Conn = RestApi(ip, port, Url, logger=logger) - Res, Errmsg, Ret = Conn.delete() - return Res, Errmsg, Ret + LocalServices = list() + # Update Service Info + for Service in ServerDetail.Services: + try: + ProcObject = Process(Service.ServiceType) + NewService = dict() + NewService['Id'] = Service.Id + NewService['GroupId'] = '' if Service.GroupId is None else Service.GroupId + NewService['Type'] = Service.ServiceType + NewService['ProcessObject'] = ProcObject + NewService['IsEnable'] = True + LocalServices.append(NewService) + + ret, errlog = isKsanServiceIdFileExists(Service.ServiceType) + if ret is False: + logger.debug('ServiceId file is not found. %s %s' % (Service.Name, errlog)) + ret, errlog = SaveKsanServiceIdFile(Service.ServiceType, Service.Id) + if ret is False: + logger.error(errlog) + else: + logger.debug('ServiceId file exists %s' % Service.Name) + #logging.log(logging.INFO, 'Service %s is added' % Service.ServiceType) + except Exception as err: + logger.error("fail to get service info %s " % str(err)) + + while True: + + if GlobalFlag['ServiceUpdated'] == Updated: + GlobalFlag['ServiceUpdated'] = Checked + #logging.log(logging.INFO,'Service Info is Updated') + break + + ServiceTypePool = list() + for Service in LocalServices: + try: + ServiceId = Service['Id'] + ProcObject = Service['ProcessObject'] + ServiceType = Service['Type'] + GroupId = Service['GroupId'] + ProcObject.GetPidWithServiceType() + ProcObject.GetUsage() + ProcObject.GetState() + Usage = UpdateServiceUsageObject() + Usage.Set(ServiceId, ProcObject.CpuUsage, ProcObject.MemoryUsed, ProcObject.ThreadCount) + Usage = jsonpickle.encode(Usage, make_refs=False, unpicklable=False) + logger.debug(Usage) + Usage = json.loads(Usage) + MqServiceUsage.Sender(Usage) + logger.debug(Usage) + except Exception as err: + logger.error("fail to get service info %s" % str(err)) + continue + + if ServiceType == TypeServiceAgent: + State = UpdateServicesStateObject() + #State.Set(ServiceId, ProcObject.State) + State.Set(ServiceId, ONLINE) + State = jsonpickle.encode(State, make_refs=False, unpicklable=False) + State = json.loads(State) + logger.debug(State) + MqServiceState.Sender(State) + ServiceStatus = ProcObject.State + #CreateServicePoolXmlFile(ServiceTypePool) + #UpdateServiceInfoDump(ServiceTypePool, ServiceId, ServiceType, GroupId, ServiceStatus) + + + root = CreateServicePoolXmlFile(ServiceTypePool) + indent(root) + tree = ET.ElementTree(root) + tree.write(ServicePoolXmlPath, encoding="utf-8", xml_declaration=True) + + time.sleep(ServiceMonitorInterval) + time.sleep(IntervalMiddle) @catch_exceptions() -def UpdateServiceGroup(Ip, Port, GroupId, Name=None, Description=None, ServiceType=None, - ServiceIds=None, addServiceIds=None, removeServiceIds=None, logger=None): - Res, Errmsg, Ret,Group = GetServiceGroup(Ip, Port, GroupId=GroupId, logger=logger) - if Res != ResOk: - return Res, Errmsg, None - else: - if Ret.Result != ResultSuccess: - return Res, Errmsg, Ret - - ExistServiceIds = GetServiceIdList(Group) - if Name is not None: - Group.Name = Name - if Description is not None: - Group.Description = Description - if ServiceType is not None: - Group.ServiceType = ServiceType - if ServiceIds is not None: - ExistServiceIds = ServiceIds - if addServiceIds is not None: - for new in addServiceIds: - if new in ExistServiceIds: - return ResDuplicateCode, ResDuplicateMsg, None - - ExistServiceIds += addServiceIds - if removeServiceIds is not None: - for ServiceId in removeServiceIds: - ExistServiceIds.remove(ServiceId) +def UpdateServiceUsage(Ip, Port, ServiceId, Usage, logger=None): - NewGroup = AddServiceGroupItems() - NewGroup.Set(Group.Name, Group.ServiceType, ServiceIpAddress=Group.ServiceIpAddress, ServiceIds=ExistServiceIds) - - Url = "/api/v1/ServiceGroups/%s" % GroupId - body = jsonpickle.encode(NewGroup, make_refs=False) - Conn = RestApi(Ip, Port, Url, params=body, logger=logger) - Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - - -def ControlSerivceGroup(Ip, Port, GroupId, Action, logger=None): - Url = "/api/v1/ServiceGroups/%s/%s" % (GroupId, Action) - body = dict() - Conn = RestApi(Ip, Port, Url, params=body, logger=logger) - Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - -def StopSerivceGroup(Ip, Port, GroupId, logger=None): - Url = "/api/v1/ServiceGroups/%s/Stop" % GroupId - body = dict() + Url = "/api/v1/Services/Usage" % ServiceId + body = jsonpickle.encode(Usage, make_refs=False) Conn = RestApi(Ip, Port, Url, params=body, logger=logger) - Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - - -def GetServiceIdList(GroupDetail): - ServiceListOfGroup = list() - for Service in GroupDetail.Services: - ServiceListOfGroup.append(Service.Id) - return ServiceListOfGroup - - -def GetServiceGroup(Ip, Port, GroupId=None, logger=None): - if GroupId is not None: - Url = "/api/v1/ServiceGroups/%s" % GroupId - ItemsHeader = False - ReturnType = ServiceGroupItemsModule - else: - Url = "/api/v1/ServiceGroups" - ItemsHeader = True - ReturnType = ResponseItemsHeaderModule - Params = dict() - Params['countPerPage'] = 100 - Conn = RestApi(Ip, Port, Url, params=Params, logger=logger) - Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) - if Res == ResOk: - AllServiceInfo = list() - if Ret.Result == ResultSuccess: - if GroupId is not None: - #GetDataFromBodyReculsive(Ret.Data, ServiceDetailModule) - #ServiceDetailInfo = jsonpickle.decode(json.dumps(Ret.Data)) - return Res, Errmsg, Ret, Ret.Data - else: - return Res, Errmsg, Ret, Ret.Data.Items - return Res, Errmsg, Ret, None - else: - return Res, Errmsg, None, None - - -def GetAllServiceGroups(Ip, Port, logger=None): - Res, Errmsg, Ret, Groups = GetServiceGroup(Ip, Port, logger=logger) - if Res == ResOk: - AllServiceGroupList = list() - if Ret.Result == ResultSuccess: - for Grp in Groups: - res, errmsg, ret, GroupDetail = GetServiceGroup(Ip, Port, GroupId=Grp.Id, logger=logger) - if res == ResOk: - if ret.Result == ResultSuccess: - AllServiceGroupList.append(GroupDetail) - else: - print('fail to get Service Group Info', ret.Message) - return Res, Errmsg, Ret, AllServiceGroupList - else: - print('fail to get Service Group List', Ret.Message) - return Res, Errmsg, Ret, None - else: - return Res, Errmsg, None, None - - -@catch_exceptions() -def ShowServiceGroup(Groups, Detail=False): - - TopTitleLine = '%s' % ('=' * 175) - GroupTitleLine = '%s' % ('-' * 175) - print(TopTitleLine) - GroupTitle ="|%s|%s|%s|%s|%s|%s|" % ('Name'.center(14), 'Description'.center(20), 'Id'.center(40), - 'ServiceType'.center(15), 'ServiceIpaddress'.center(20), ' ' * 59) - print(GroupTitle) - print(TopTitleLine) - - ServiceTitleLine = '%s%s' % (' ' *10, '-' * 165) - for Grp in Groups: - _grp = "|%s|%s|%s|%s|%s|%s|" % ( Grp.Name.center(14), str(Grp.Description).center(20), Grp.Id.center(40), - Grp.ServiceType.center(15), str(Grp.ServiceIpAddress).center(20), ' ' * 59) - - print(_grp) - print(GroupTitleLine) - if Detail: - ServiceTitle = "%s|%s|%s|%s|%s|%s|%s|%s|%s|" % (' ' * 10, 'Service Name'.center(16), 'Description'.center(20), - 'Service Id'.center(40), 'State'.center(15), 'Service Type'.center(15), - 'Cpu Usage'.center(20), 'Memoy Used'.center(20), 'Thread Cnt'.center(10)) - print(ServiceTitle) - if len(Grp.Services) >= 0: - print(ServiceTitleLine) - else: - print(GroupTitleLine) - for idx, svc in enumerate(Grp.Services): - _svc ="%s|%s|%s|%s|%s|%s|%s|%s|%s|" % \ - (' ' * 10, svc.Name.center(16), '{:20.20}'.format(str(svc.Description).center(20)), - svc.Id.center(40), svc.State.center(15), str(svc.ServiceType).center(15), - str(svc.CpuUsage).center(20), str(svc.MemoryUsed).center(20), str(svc.ThreadCount).center(10)) - - print(_svc) - if len(Grp.Services) -1 == idx: - print(GroupTitleLine) - else: - print(ServiceTitleLine) - - -@catch_exceptions() -def GetSeviceConfStringForDisplay(Conf, Prefix, ConfFile=None): - Conf = json.loads(Conf) - StrConf = '' - if ConfFile is None: - for key1, val1 in Conf.items(): - if isinstance(val1, dict): - for key2, val2 in val1.items(): - ConfLineStr = "%s_%s=%s" % (key1, key2, str(val2)) - StrConf += "%s%s|\n" % (Prefix, ConfLineStr.ljust(100-13)) - else: - ConfLineStr = "%s=%s" % (key1, str(val1)) - StrConf += '%s%s|\n' % (Prefix, ConfLineStr.ljust(100-13)) - else: - for key1, val1 in Conf.items(): - if isinstance(val1, dict): - for key2, val2 in val1.items(): - StrConf += "%s_%s=%s\n" % (key1, key2, str(val2)) - else: - StrConf += "%s=%s\n" % (key1, str(val1)) - - return StrConf - -@catch_exceptions() -def GetServiceConfString(Conf): - Conf = json.loads(Conf) - StrConf = '' - for key1, val1 in Conf.items(): - if isinstance(val1, dict): - for key2, val2 in val1.items(): - StrConf += "%s_%s=%s" % (key1, key2, str(val2)) - else: - StrConf += "%s=%s" % (key1, str(val1)) - return StrConf - - -@catch_exceptions() -def GetDspConfString(Conf, Prefix): - Conf = json.loads(Conf) - StrConf = '' - for key1, val1 in Conf.items(): - if isinstance(val1, dict): - for key2, val2 in val1.items(): - ConfLineStr = "%s_%s=%s" % (key1, key2, str(val2)) - StrConf += "%s%s|\n" % (Prefix, ConfLineStr.ljust(100-13)) - else: - ConfLineStr = "%s=%s" % (key1, str(val1)) - StrConf += '%s%s|\n' % (Prefix, ConfLineStr.ljust(100-13)) - return StrConf - - -@catch_exceptions() -def GetServiceConfigList(PortalIp, PortalPort, ApiKey, ServiceType, logger=None): - """ - Get config list from portal - :param PortalIp: - :param PortalPort: - :param ServiceType: - :param logger: - :return: - """ - Url = '/api/v1/Config/List/%s' % ServiceType - ItemsHeader = True - ReturnType = ResponseItemsHeaderModule - Params = dict() - Params['countPerPage'] = 100 - - Conn = RestApi(PortalIp, PortalPort, Url, authkey=ApiKey, params=Params, logger=logger) - Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) - if Res == ResOk: - ConfigList = list() - if Ret.Result == ResultSuccess: - return Res, Errmsg, Ret, Ret.Data - else: - return Res, Errmsg, Ret, None - else: - return Res, Errmsg, None, None - -@catch_exceptions() -def LoadServiceList(conf, ServiceList, LocalIpList, logger): - Res, Errmsg, Ret, Data = GetServerInfo(conf.PortalHost, conf.PortalPort, conf.PortalApiKey, conf.ServerId, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - for service in Data.Services: - ServiceList['IdList'].append(service.Id) - ServiceList['Details'][service.Id] = service - for Interface in Data.NetworkInterfaces: - LocalIpList.append(Interface.IpAddress) - print(ServiceList) - print(LocalIpList) - - -@catch_exceptions() -def ShowConfigList(ConfigList, Detail=False): - """ - Display all config info with type - :param ConfigList: - :param Detail: - :return: - """ - print(ConfigCompartLine) - for Config in ConfigList.Items: - DsPServiceConf(Config) - -def DsPServiceConf(Config, TopTitleLine=False, ConfFile=None): - """ - Display service config with type - :param Config: - :param TopTitleLine: - :return: - """ - Type = Config.Type - Version = Config.Version - Conf = Config.Config - ConfPartPrefix = '|%s|' % (' ' * 10) - RegiDate = Config.RegDate - StrConf = GetSeviceConfStringForDisplay(Conf, ConfPartPrefix, ConfFile=ConfFile) - if ConfFile is None: - if StrConf is None: - print('Invalid Conf. version:%d \n%s' % (Version, str(Conf))) - else: - #if TopTitleLine is not True: - # print(ConfigCompartLine) - ConfigStr = """|%s|%s| -%s -|%s|%s| -%s -|%s|%s| -%s -|%s|%s| -%s%s - """ % ('Type'.center(10), Type.center(ConfigMaxValueLen), ConfigTitleLine, 'Version'.center(10), - str(Version).center(ConfigMaxValueLen), ConfigTitleLine, 'Date'.center(10), RegiDate.center(ConfigMaxValueLen), - ConfigTitleLine, - 'Conf'.center(10), ' ' * (100 - 13), StrConf.center(ConfigMaxValueLen), ConfigCompartLine) - print(ConfigStr) - else: - with open(ConfFile, 'w') as f: - f.write(StrConf) - - -@catch_exceptions() -def GetServiceConfig(PortalIp, PortalPort, ApiKey, ServiceType, Version=None, logger=None): - Url = '/api/v1/Config/%s' % ServiceType - if Version is not None: - Url += '/%s' % Version - - ItemsHeader = False - ReturnType = ServiceGroupItemsModule - - Params = dict() - Conn = RestApi(PortalIp, PortalPort, Url, authkey=ApiKey, params=Params, logger=logger) - Res, Errmsg, Ret = Conn.get(ItemsHeader=ItemsHeader, ReturnType=ReturnType) - if Res == ResOk: - AllServiceInfo = list() - if Ret.Result == ResultSuccess: - return Res, Errmsg, Ret, Ret.Data - else: - return Res, Errmsg, Ret, None - else: - return Res, Errmsg, None, None - -@catch_exceptions() -def UpdateServiceConfigVersion(PortalIp, PortalPort, ApiKey, ServiceType, Version, logger=None): - """ - Update config version - :param PortalIp: - :param PortalPort: - :param ServiceType: - :param ConfigFilePath: - :param logger: - :return: - """ - - Url = "/api/v1/Config/%s/%s" % (ServiceType, Version) - body = json.dumps(dict()) - Conn = RestApi(PortalIp, PortalPort, Url, authkey=ApiKey, params=body, logger=logger) Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) return Res, Errmsg, Data -@catch_exceptions() -def SetServiceConfig(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, ConfFile=None, logger=None): - # Get Current Service config - Ret, Conf = GetConfig(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger, ConfFile=ConfFile) - if Ret is False: - return Ret, Conf, None - - Url = "/api/v1/Config/%s" % ServiceType - body = Conf - Conn = RestApi(IfsPortalIp, IfsPortalPort, Url, authkey=ApiKey, params=body, logger=logger) - Res, Errmsg, Data = Conn.post(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - - -@catch_exceptions() -def UpdateServiceConfig(ServiceType, Version=None, ConfigFile=False): - pass - -@catch_exceptions() -def RemoveServiceConfig(PortalIp, PortalPort, ApiKey, ServiceType, Version, logger=None): - """ - delete service config from config list - :param ip: - :param port: - :param Id: - :param logger: - :return:tuple(error code, error msg, ResponseHeader class) - """ - Url = '/api/v1/Config/%s/%s' % (ServiceType, Version) - Conn = RestApi(PortalIp, PortalPort, Url, authkey=ApiKey, logger=logger) - Res, Errmsg, Ret = Conn.delete() - return Res, Errmsg, Ret - - -def GetConfig(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger, ConfFile=None): - if ConfFile is None: - Conf = GetConfigFromUser(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger) - Conf = json.dumps(Conf) - Conf = Conf.replace('"', '\\"') - Conf = '"'+ Conf + '"' - return True, Conf - else: - if os.path.exists(ConfFile): - Ret, Conf = GetConf(ConfFile, FileType='non-ini', ReturnType='dict') - if Ret is True: - Conf = json.dumps(Conf) - Conf = Conf.replace('"', '\\"') - Conf = '"' + Conf + '"' - return True, Conf - else: - return Ret, Conf - else: - return False, '%s is not found' % ConfFile - - - - -def GetConfigFromUser(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger): - - ConfigInfo = GetServiceCurrentConfigInfo(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, logger) - - Conf = dict() - for info in ConfigInfo: - ConfKey = info['key'] - QuestionString = info['question'] - ValueType = info['type'] - DefaultValue = info['value'] - if 'valid_answer_list' in info: - ValueAnswerList = info['valid_answer_list'] - else: - ValueAnswerList = None - if 'Activate' in info: - if info['Activate'] is False: - Conf[ConfKey] = info['value'] - continue - Conf[ConfKey]= get_input(QuestionString, ValueType, DefaultValue, ValidAnsList=ValueAnswerList) - return Conf - - -def GetServiceCurrentConfigInfo(PortalIp, PortalPort, PortalApiKey, ServiceType, logger): - # Get MqConfig - """ - Get Current Config if exists, or Default Config if not exists. - :param PortalIp: - :param PortalPort: - :param PortalApiKey: - :param ServiceType: - :param logger: - :return: - """ - - Res, Errmsg, Ret, Data = GetServiceConfig(PortalIp, PortalPort, PortalApiKey, ServiceType, logger=logger) - if Res != ResOk: - logger.error('fail to get config %s' % Errmsg) - else: - if (Ret.Result == ResultSuccess): - DsPServiceConf(Data) - else: - logger.error('fail to get config %s' % Ret.Message) - - ConfigInfo = None - if ServiceType == TypeServiceOSD: - if Data: - CurrentConf = json.loads(Data.Config) - for info in OsdDefaultConfigInfo: - try: - ConfKey = info['key'] - info['value'] = CurrentConf[ConfKey] - except Exception as err: - pass - ConfigInfo = OsdDefaultConfigInfo - - elif ServiceType == TypeServiceGW: - - if Data: - CurrentConf = json.loads(Data.Config) - for info in S3DefaultConfigInfo: - try: - ConfKey = info['key'] - info['value'] = CurrentConf[ConfKey] - except Exception as err: - pass - else: - - # Get RaggitMq Config - Ret, MqConfig = GetBaseConfig(PortalIp, PortalPort, PortalApiKey, TypeServiceRabbitMq, logger) - if Ret is False: - return None - CurrentMqConfig = json.loads(MqConfig.Config) - - # Get MariaDB Config - Ret, MariaDBConfig = GetBaseConfig(PortalIp, PortalPort, PortalApiKey, TypeServiceMariaDB, logger) - if Ret is False: - return None - - CurrentMariaDBConfig = json.loads(MariaDBConfig.Config) - - UpdateGwBaseConfig(S3DefaultConfigInfo, CurrentMqConfig, CurrentMariaDBConfig) - - ConfigInfo = S3DefaultConfigInfo - - elif ServiceType == TypeServiceMongoDB: - if Data: - CurrentConf = json.loads(Data.Config) - for info in OsdDefaultConfigInfo: - try: - ConfKey = info['key'] - if ConfKey == 'objM.mq_host': - info['value'] = CurrentConf['Host'] - elif ConfKey == 'objM.mq_osd_exchange_name': - info['value'] = CurrentConf['ExchangeName'] - except Exception as err: - pass - ConfigInfo = MongoDbConfigInfo - - elif ServiceType == TypeServiceLifecycle: - if Data: - CurrentConf = json.loads(Data.Config) - for info in LifecycleDefaultConfigInfo: - try: - ConfKey = info['key'] - info['value'] = CurrentConf[ConfKey] - except Exception as err: - pass - ConfigInfo = LifecycleDefaultConfigInfo - - - return ConfigInfo - -def GetBaseConfig(PortalIp, PortalPort, PortalApiKey, ConfigType, logger ): - """ - Get Base(Mariadb, RabbitMq) Config - :param PortalIp: - :param PortalPort: - :param PortalApiKey: - :param ConfigType: - :param logger: - :return: - """ - - Res, Errmsg, Ret, MqConfig = GetServiceConfig(PortalIp, PortalPort, PortalApiKey, ConfigType, logger=logger) - if Res != ResOk: - logger.error('fail to get %s config %s' % (ConfigType, Errmsg)) - else: - if (Ret.Result == ResultSuccess): - #DsPServiceConf(MqConfig) - return True, MqConfig - else: - logger.error('fail to get %s config %s' % (ConfigType, Ret.Message)) - return False, None - -def UpdateGwBaseConfig(GwConfig, MqConfig, MariaDBConfig): - """ - Merge GwConfig and MqConfig. - :param GwConfig: - :param MqConfig: - :return: - """ - MqKeyMap = {"objM.mq_host": "Host", "objM.mq_exchange_name": "ExchangeName"} - MariaDBKeyMap = {"gw.db_host": "Host", "gw.db_name": "Name", "gw.db_port": "Port", "gw.db_user": "User", - "gw.db_password": "Password", "objM.db_host": "Host", "objM.db_name": "Name", "objM.db_port": "Port", - "objM.db_user=root": "User", "objM.db_password": "Password"} - - - for info in GwConfig: - try: - ConfKey = info['key'] - if ConfKey in MqKeyMap: - BaseKey = MqKeyMap[ConfKey] - info['value'] = MqConfig[BaseKey] - elif ConfKey in MariaDBKeyMap: - BaseKey = MariaDBKeyMap[ConfKey] - info['value'] = MariaDBConfig[BaseKey] - - except Exception as err: - print('fail to update ksanGW Mq & MariaDB Config %s' % str(err)) - -def isKsanServiceIdFileExists(ServiceType): - if ServiceType in ServiceTypeServiceHiddenPathMap: - isDockerService = False - - if ServiceType in ServiceTypeSystemdServiceMap: - SystemdServiceUnitPath = ServiceTypeSystemdServiceMap[ServiceType] - with open('/etc/systemd/system/'+ SystemdServiceUnitPath, 'r') as f: - SystemdServiceContens = f.read() - if 'Requires=docker.service' in SystemdServiceContens: - isDockerService = True - - - ServiceHiddenPath = ServiceTypeServiceHiddenPathMap[ServiceType] - - if isDockerService is False: - if os.path.exists(ServiceHiddenPath): - return True - else: - return False - else: - CopiedServiceIdFile = '/tmp/.TmpServiceId%d' % int(time.time()) - ServiceContainerName = ServiceTypeDockerServiceContainerNameMap[ServiceType] - DockerGetServiceFileCmd = 'docker cp %s:%s %s' % (ServiceContainerName, ServiceHiddenPath, CopiedServiceIdFile) - shcall(DockerGetServiceFileCmd) - if os.path.exists(CopiedServiceIdFile): - return True - else: - return False +def CreateServicePoolXmlFile(ServiceTypePool): + root = ET.Element('SERVICELIST') + for Type in ServiceTypePool: + TypeTag = CreateTypeElement(Type['ServiceType']) + for Service in Type['ServiceList']: + ServiceTag = CreateServiceElement(Service['ServiceId'], Service['GroupId'], Service['Status']) + TypeTag.append(ServiceTag) + root.append(TypeTag) + return root + + +def CreateTypeElement(ServiceType): + NewTypeTag = ET.Element("SERVICETYPE") + # last_updated의 text를 지정한다 + # NewDiskTag.text = '\n\t' + NewTypeTag.attrib["type"] = ServiceType + return NewTypeTag + + +def CreateServiceElement(ServiceId, GroupId, Status): + NewServiceTag = ET.Element("SERVICE") + # last_updated의 text를 지정한다 + # NewDiskTag.text = '\n\t' + NewServiceTag.attrib["id"] = ServiceId + NewServiceTag.attrib["group"] = GroupId + NewServiceTag.attrib["status"] = Status + return NewServiceTag + + +def indent(elem, level=0): + i = "\n" + level*" " + if len(elem): + if not elem.text or not elem.text.strip(): + elem.text = i + " " + if not elem.tail or not elem.tail.strip(): + elem.tail = i + for elem in elem: + indent(elem, level+1) + if not elem.tail or not elem.tail.strip(): + elem.tail = i else: - return True - - -def SaveKsanServiceIdFile(ServiceType, ServiceId): - if ServiceType in ServiceTypeServiceHiddenPathMap: - ServiceHiddenPath = ServiceTypeServiceHiddenPathMap[ServiceType] - isDockerService = False - - if ServiceType in ServiceTypeSystemdServiceMap: - SystemdServiceUnitPath = ServiceTypeSystemdServiceMap[ServiceType] - with open('/etc/systemd/system/'+ SystemdServiceUnitPath, 'r') as f: - SystemdServiceContens = f.read() - if 'Requires=docker.service' in SystemdServiceContens: - isDockerService = True - - with open(ServiceHiddenPath, 'w') as f: - f.write(ServiceId) - f.flush() - - if isDockerService is True: - SystemdServiceUnitPath = ServiceTypeSystemdServiceMap[ServiceType] - ServiceStopCmd = 'systemctl stop %s' % SystemdServiceUnitPath - shcall(ServiceStopCmd) - ServiceContainerName = ServiceTypeDockerServiceContainerNameMap[ServiceType] - DockerCopyCmd = 'docker cp %s %s:%s' % (ServiceHiddenPath, ServiceContainerName, ServiceHiddenPath) - shcall(DockerCopyCmd) - os.unlink(ServiceHiddenPath) - + if level and (not elem.tail or not elem.tail.strip()): + elem.tail = i -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: service_manage.py {add|delete|start|stop|set|list} [option] - add -S -N -G -T : Add disk - delete -I : Delete Service - set -I [Rw|Ro] : Set disk mode. Ro:ReadOnly, Rw: ReadWrite - start -I : Start disk - stop -I : Stop disk - list : Display disk info - [options] - -a : show all server info - -T : show information in detail - -h, --help : show this help message and exit -""" - print(Usage) - -""" -if __name__ == '__main__': - usage = "Usage: %prog {init|add|delete|update|list|status} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-I', "--Id", dest="ServerId", help='server id') - parser.add_option('-N', "--Name", dest="Name", default='', help='server description') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='debug mode') - parser.add_option('-G', "--GroupId", dest="GroupId", help='Group Id') - parser.add_option('-T', "--ServiceType", dest="ServiceType", help='Service Type') - parser.add_option('-V', "--VlanId", dest="VlanId", help='Vlan Id') - parser.add_option('-l', "--Detail", dest="Detail", action='store_true', help='server info in detail') - - options, args = parser.parse_args() - if len(args) != 1: - parser.print_help() - sys.exit(-1) - IfsPortalIp = '127.0.0.1' - IfsPortalPort = 5443 - IfsMqPort = 5672 - ret, conf = read_conf(MonServicedConfPath) - if ret is True: - IfsPortalIp = conf['mgs']['MgsIp'] - IfsPortalPort = int(conf['mgs']['IfsPortalPort']) - IfsMqPort = int(conf['mgs']['MqPort']) +def UnknownServiceRestart(ServiceList): + ServiceBinMap = dict() + ServiceBinMap[KsanGWName] = 'systemctl restart %s.service' % KsanGWName + ServiceBinMap[KsanOSDName] = 'systemctl restart %s.service' % KsanOSDName + ServiceBinMap[KsanLifecycleManagerName] = 'systemctl restart %s.service' % KsanLifecycleManagerName + ServiceBinMap[KsanLogManagerName] = 'systemctl restart %s.service' % KsanLogManagerName + ServiceBinMap[KsanReplicationManagerName] = 'systemctl restart %s.service' % KsanReplicationManagerName + ServiceBinMap[KsanAgentName] = '/usr/local/ksan/sbin/ksanAgent start' - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - - if args[0] == 'add': - Res, Errmsg, Ret = AddService(IfsPortalIp, IfsPortalPort, options.Name, options.ServiceType, options.ServerId, - options.GroupId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'delete': - Res, Errmsg, Ret = DeleteService(IfsPortalIp, IfsPortalPort, options.ServerId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'list': - Res, Errmsg, Ret, Data = GetServiceInfo(IfsPortalIp, IfsPortalPort,logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - ShowServiceInfo(Data, Detail=options.Detail) - else: - print(Errmsg) -""" + for Svc in ServiceList: + if Svc.State in ['Unknown', 'Offline', 'Timeout']: + Cmd = ServiceBinMap[Svc.ServiceType] + out, err = shcall(Cmd) + print(out, err) @@ -1281,18 +281,23 @@ def ServiceUtilHandler(Conf, Action, Parser, logger): else: print(Errmsg) elif Action.lower() == 'list': - if options.Detail: - Res, Errmsg, Ret, Data = GetAllServerDetailInfo(PortalIp, PortalPort, PortalApiKey,logger=logger) - else: - Res, Errmsg, Ret, Data = GetServiceInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) + #if options.Detail: + #Res, Errmsg, Ret, Data = GetAllServerDetailInfo(PortalIp, PortalPort, PortalApiKey,logger=logger) + #else: + Res, Errmsg, Ret, Data = GetServiceInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) if Res == ResOk: if Ret.Result == ResultSuccess: - if options.Detail: - ShowServiceInfoWithServerInfo(Data, Detail=options.Detail) - #ShowServiceInfo(Data, Detail=options.Detail) + if options.MoreDetail: + Detail = MoreDetailInfo + elif options.Detail: + Detail = DetailInfo else: - ShowServiceInfo(Data, Detail=options.Detail) + Detail = SimpleInfo + ShowServiceInfoWithServerInfo(Data, Detail=Detail) + #ShowServiceInfo(Data, Detail=options.Detail) + #else: + # ShowServiceInfo(Data, Detail=options.Detail) else: print(Ret.Result, Ret.Message) else: @@ -1339,6 +344,24 @@ def ServiceUtilHandler(Conf, Action, Parser, logger): else: print(Errmsg) + elif Action.lower() == 'unknown_restart': + #if options.Detail: + #Res, Errmsg, Ret, Data = GetAllServerDetailInfo(PortalIp, PortalPort, PortalApiKey,logger=logger) + #else: + Res, Errmsg, Ret, Data = GetServiceInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) + + if Res == ResOk: + if Ret.Result == ResultSuccess: + UnknownServiceRestart(Data) + #ShowServiceInfo(Data, Detail=options.Detail) + #else: + # ShowServiceInfo(Data, Detail=options.Detail) + else: + print(Ret.Result, Ret.Message) + else: + print(Errmsg) + else: Parser.print_help() + diff --git a/core/common/service/service_mq_handle.py b/core/common/service/service_mq_handle.py deleted file mode 100644 index b04378d3..00000000 --- a/core/common/service/service_mq_handle.py +++ /dev/null @@ -1,467 +0,0 @@ -#!/usr/bin/env python -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import psutil -import os, sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from service.service_manage import * -#import ksan.service.s3 as S3 -from service.osd import KsanOsd -from common.utils import IsDaemonRunning -from common.init import WaitAgentConfComplete -from const.mq import RoutKeyServiceState, MqVirtualHost, ExchangeName, RoutKeyServiceControlFinder, \ - RoutKeyServiceUsage -from const.service import UpdateServicesStateObject, SystemdServicePidFinder -from const.common import ServiceTypeSystemdServiceMap -from service.gw import KsanGW -from service.monitor import KsanMonitor -from service.mongo import KsanMongoDB -from service.control import ServiceUnit -import mqmanage.mq -import logging -import xml.etree.ElementTree as ET -import inspect - -@catch_exceptions() -def MqServiceHandler(MonConf, RoutingKey, Body, Response, ServerId, ServiceList, LocalIpList, GlobalFlag, logger): - """ - Message Queue Service Handler - """ - logger.debug("%s %s %s" % (str(RoutingKey), str(Body), str(Response))) - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Body = Body.decode('utf-8') - Body = json.loads(Body) - body = DictToObject(Body) - - # Service Control Handle - if RoutKeyServiceControlFinder.search(RoutingKey): - logging.log(logging.INFO, "%s" % str(Body)) - #if body.Id not in ServiceList['IdList']: - # LoadServiceList(MonConf, ServiceList, LocalIpList, logger) - - Response.IsProcessed = True - #if body.Id not in ServiceList['Details']: - # logger.error('Invalid Service Id %s' % body.Id) - # return ResponseReturn - ServiceType = body.ServiceType - #if ServiceType == TypeHaproxy: - # Ret = Hap.StartStop(Body) - #elif ServiceType == TypeS3: - # Ret = S3.StartStop(Body) - service = ServiceUnit(logger, ServiceType) - if body.Control == START: - Ret, ErrMsg = service.Start() - elif body.Control == STOP: - Ret, ErrMsg = service.Stop() - elif body.Control == RESTART: - Ret, ErrMsg = service.Restart() - elif ServiceType.lower() == TypeServiceAgent.lower(): - Ret = True - ErrMsg = '' - else: - Ret = False - ErrMsg = 'Invalid Serivce Type' - - if Ret is False: - ResponseReturn = mqmanage.mq.MqReturn(Ret, Code=1, Messages=ErrMsg) - logger.error(ResponseReturn) - else: - logging.log(logging.INFO, ResponseReturn) - return ResponseReturn - # Serivce Config Handle - elif ".config." in RoutingKey: - if body.Id in ServiceList['IdList']: - Response.IsProcessed = True - logger.debug(ResponseReturn) - return ResponseReturn - elif RoutingKey.endswith((".added", ".updated", ".removed")): - ServiceType = body.ServiceType - GlobalFlag['ServiceUpdated'] = Updated - - if RoutingKey.endswith(".added"): - if isKsanServiceIdFileExists(ServiceType) is False: - logger.debug('ServiceId file is not found. %s' % body.Id) - SaveKsanServiceIdFile(ServiceType, body.Id) - - Response.IsProcessed = True - logger.debug(ResponseReturn) - return ResponseReturn - else: - Response.IsProcessed = True - return ResponseReturn - - -def Old_MqServiceHandler(MonConf, RoutingKey, Body, Response, ServerId, ServiceList, LocalIpList, GlobalFlag, logger): - """ - Message Queue Service Handler - """ - logger.debug("%s %s %s" % (str(RoutingKey), str(Body), str(Response))) - ResponseReturn = mqmanage.mq.MqReturn(ResultSuccess) - Body = Body.decode('utf-8') - Body = json.loads(Body) - body = DictToObject(Body) - - # Service Control Handle - if RoutKeyServiceControlFinder.search(RoutingKey): - logging.log(logging.INFO, "%s" % str(Body)) - #if body.Id not in ServiceList['IdList']: - # LoadServiceList(MonConf, ServiceList, LocalIpList, logger) - - Response.IsProcessed = True - #if body.Id not in ServiceList['Details']: - # logger.error('Invalid Service Id %s' % body.Id) - # return ResponseReturn - ServiceType = body.ServiceType - #if ServiceType == TypeHaproxy: - # Ret = Hap.StartStop(Body) - #elif ServiceType == TypeS3: - # Ret = S3.StartStop(Body) - if ServiceType == TypeServiceOSD: - Osd = KsanOsd(logger) - if body.Control == START: - Ret, ErrMsg = Osd.Start() - elif body.Control == STOP: - Ret, ErrMsg = Osd.Stop() - elif body.Control == RESTART: - Ret, ErrMsg = Osd.Restart() - else: - Ret = False - ErrMsg = 'Invalid Control Code' - elif ServiceType == TypeServiceGW: - Gw = KsanGW(logger) - if body.Control == START: - Ret, ErrMsg = Gw.Start() - elif body.Control == STOP: - Ret, ErrMsg = Gw.Stop() - elif body.Control == RESTART: - Ret, ErrMsg = Gw.Restart() - else: - Ret = False - ErrMsg = 'Invalid Control Code' - elif ServiceType == TypeServiceMongoDB: - mongo = KsanMongoDB(logger) - Ret, ErrMsg = mongo.GetMongoDBConf() - if Ret is True: - if body.Control == START: - Ret, ErrMsg = mongo.Start() - elif body.Control == STOP: - Ret, ErrMsg = mongo.Stop() - elif body.Control == RESTART: - Ret, ErrMsg = mongo.Restart() - else: - Ret = False - ErrMsg = 'Invalid Control Code' - - - elif ServiceType == TypeServiceMonitor: - Monitor = KsanMonitor(logger) - if body.Control == START: - Ret, ErrMsg = Monitor.Start() - elif body.Control == STOP: - Ret, ErrMsg = Monitor.Stop() - elif body.Control == RESTART: - Ret, ErrMsg = Monitor.Restart() - else: - Ret = False - ErrMsg = 'Invalid Control Code' - elif ServiceType == TypeServiceAgent: - Ret = True - ErrMsg = '' - else: - Ret = False - ErrMsg = 'Invalid Serivce Type' - - if Ret is False: - ResponseReturn = mqmanage.mq.MqReturn(Ret, Code=1, Messages=ErrMsg) - logger.error(ResponseReturn) - else: - logging.log(logging.INFO, ResponseReturn) - return ResponseReturn - # Serivce Config Handle - elif ".config." in RoutingKey: - if body.Id in ServiceList['IdList']: - Response.IsProcessed = True - logger.debug(ResponseReturn) - return ResponseReturn - elif RoutingKey.endswith((".added", ".updated", ".removed")): - GlobalFlag['ServiceUpdated'] = Updated - Response.IsProcessed = True - logger.debug(ResponseReturn) - return ResponseReturn - else: - Response.IsProcessed = True - return ResponseReturn - - - - -class Process: - def __init__(self,ServiceType): - self.ServiceType = ServiceType - self.SystemdServiceName = ServiceTypeSystemdServiceMap[ServiceType] if ServiceType in ServiceTypeSystemdServiceMap else None - self.Pid = None - self.CpuUsage = 0 - self.MemoryUsed = 0 - self.ThreadCount = 0 - self.State = 'Offline' - self.GetPidWithServiceType() - self.GetState() - self.GetUsage() - - def GetPidWithServiceType(self): - if self.SystemdServiceName is not None: - SystemdCmd = 'systemctl status %s' % self.SystemdServiceName - out, err = shcall(SystemdCmd) - Pid = SystemdServicePidFinder.search(out) - if not Pid: - self.Pid = None - else: - self.Pid = int(Pid.groups()[0]) - else: - if self.ServiceType == TypeServiceAgent: - Ret, Pid = IsDaemonRunning(KsanAgentPidFile, CmdLine='ksanAgent') - if Ret is True: - self.Pid = int(Pid) - else: - self.Pid = None - - def OldGetPidWithServiceType(self): - if self.ServiceType == TypeServiceGW: - Ret, Pid = IsDaemonRunning(KsanGwPidFile, CmdLine='ksan-gw') - if Ret is True: - self.Pid = int(Pid) - else: - self.Pid = None - - elif self.ServiceType == TypeServiceOSD: - Ret, Pid = IsDaemonRunning(KsanOsdPidFile, CmdLine='ksan-osd') - if Ret is True: - self.Pid = int(Pid) - else: - self.Pid = None - elif self.ServiceType == TypeServiceAgent: - Ret, Pid = IsDaemonRunning(KsanAgentPidFile, CmdLine='ksanAgent') - if Ret is True: - self.Pid = int(Pid) - else: - self.Pid = None - elif self.ServiceType == TypeServiceMonitor: - Ret, Pid = IsDaemonRunning(KsanMonitorPidFile, CmdLine='ksanMonitor') - if Ret is True: - self.Pid = int(Pid) - else: - self.Pid = None - - elif self.ServiceType == TypeServiceMongoDB: - Ret, Pid = IsDaemonRunning(KsanMongosPidFile, CmdLine='mongos') - if Ret is True: - self.Pid = int(Pid) - else: - self.Pid = None - - else: - pass - - def GetUsage(self): - try: - p = psutil.Process(pid=self.Pid) - self.CpuUsage = p.cpu_percent(interval=1) - m = p.memory_full_info() - #self.MemoryUsed = m.rss + m.vms + m.shared + m.data - self.MemoryUsed = m.rss - self.ThreadCount = p.num_threads() - except Exception as err: - self.CpuUsage = 0 - self.MemoryUsed = 0 - self.ThreadCount = 0 - - - def GetState(self): - if self.Pid is not None: - self.State = ONLINE - else: - self.State = OFFLINE - - - -@catch_exceptions() -def ServiceMonitoring(Conf, GlobalFlag, logger): - - Conf = WaitAgentConfComplete(inspect.stack()[1][3], logger, CheckNetworkDevice=True, CheckNetworkId=True) - - conf = GetAgentConfig(Conf) - - KsanServiceRegister(conf, TypeServiceAgent, logger) - - MqServiceUsage = mqmanage.mq.Mq(conf.MQHost, int(conf.MQPort), MqVirtualHost, conf.MQUser, conf.MQPassword, - RoutKeyServiceUsage, ExchangeName, QueueName='') - MqServiceState = mqmanage.mq.Mq(conf.MQHost, int(conf.MQPort), MqVirtualHost, conf.MQUser, conf.MQPassword, - RoutKeyServiceState, ExchangeName, QueueName='') - - ServiceMonitorInterval = int(conf.ServiceMonitorInterval)/1000 - # local service list - LocalServices = list() # [{ 'Id': ServiceId, 'Type': 'Osd', 'ProcessObject':Object, 'IsEnable': True, 'GroupId': GroupId, 'Status': 'Online'}] - while True: - Res, Errmgs, Ret, ServerDetail = GetServerInfo(conf.PortalHost, conf.PortalPort, conf.PortalApiKey, conf.ServerId, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - LocalServices = list() - # Update Service Info - for Service in ServerDetail.Services: - try: - ProcObject = Process(Service.ServiceType) - NewService = dict() - NewService['Id'] = Service.Id - NewService['GroupId'] = '' if Service.GroupId is None else Service.GroupId - NewService['Type'] = Service.ServiceType - NewService['ProcessObject'] = ProcObject - NewService['IsEnable'] = True - LocalServices.append(NewService) - - if isKsanServiceIdFileExists(Service.ServiceType) is False: - logger.debug('ServiceId file is not found. %s' % Service.Name) - SaveKsanServiceIdFile(Service.ServiceType, Service.Id) - #logging.log(logging.INFO, 'Service %s is added' % Service.ServiceType) - except Exception as err: - logger.error("fail to get service info %s " % str(err)) - - while True: - - if GlobalFlag['ServiceUpdated'] == Updated: - GlobalFlag['ServiceUpdated'] = Checked - #logging.log(logging.INFO,'Service Info is Updated') - break - - ServiceTypePool = list() - for Service in LocalServices: - try: - ServiceId = Service['Id'] - ProcObject = Service['ProcessObject'] - ServiceType = Service['Type'] - GroupId = Service['GroupId'] - ProcObject.GetPidWithServiceType() - ProcObject.GetUsage() - ProcObject.GetState() - Usage = UpdateServiceUsageObject() - Usage.Set(ServiceId, ProcObject.CpuUsage, ProcObject.MemoryUsed, ProcObject.ThreadCount) - Usage = jsonpickle.encode(Usage, make_refs=False, unpicklable=False) - logger.debug(Usage) - Usage = json.loads(Usage) - MqServiceUsage.Sender(Usage) - logger.debug(Usage) - except Exception as err: - logger.error("fail to get service info %s" % str(err)) - continue - - if ServiceType == TypeServiceAgent: - State = UpdateServicesStateObject() - State.Set(ServiceId, ProcObject.State) - State = jsonpickle.encode(State, make_refs=False, unpicklable=False) - State = json.loads(State) - logger.debug(State) - MqServiceState.Sender(State) - ServiceStatus = ProcObject.State - #CreateServicePoolXmlFile(ServiceTypePool) - #UpdateServiceInfoDump(ServiceTypePool, ServiceId, ServiceType, GroupId, ServiceStatus) - - - root = CreateServicePoolXmlFile(ServiceTypePool) - indent(root) - tree = ET.ElementTree(root) - tree.write(ServicePoolXmlPath, encoding="utf-8", xml_declaration=True) - - time.sleep(ServiceMonitorInterval) - time.sleep(IntervalMiddle) - - -@catch_exceptions() -def UpdateServiceUsage(Ip, Port, ServiceId, Usage, logger=None): - - Url = "/api/v1/Services/Usage" % ServiceId - body = jsonpickle.encode(Usage, make_refs=False) - Conn = RestApi(Ip, Port, Url, params=body, logger=logger) - Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - - -@catch_exceptions() -def UpdateServiceState(Ip, Port, ServiceId, State, logger=None): - - Url = "/api/v1/Services/State" % ServiceId - service = UpdateServicesStateObject() - service.Set(ServiceId, State) - body = jsonpickle.encode(service, make_refs=False) - Conn = RestApi(Ip, Port, Url, params=body, logger=logger) - Res, Errmsg, Data = Conn.put(ItemsHeader=False, ReturnType=ResponseHeaderModule) - return Res, Errmsg, Data - -def UpdateServiceInfoDump(ServiceTypePool, ServiceId, ServiceType, GroupId, Status): - """ - ServicePool: Current enabled service list [{"ServiceType": type, - "List": [{"ServiceId": Id, "GroupId": GroupId, "Status": "Online"}}]}] - - """ - for Service in ServiceTypePool: - if Service['ServiceType'] == ServiceType: - NewService = {"ServiceId": ServiceId, "GroupId": GroupId, "Status": Status} - ServiceTypePool.append(NewService) - return - NewType = {"ServiceType": ServiceType, "ServiceList": []} - NewService = {"ServiceId": ServiceId, "GroupId": GroupId, "Status": Status} - NewType["ServiceList"].append(NewService) - ServiceTypePool.append(NewType) - return - -def CreateServicePoolXmlFile(ServiceTypePool): - root = ET.Element('SERVICELIST') - for Type in ServiceTypePool: - TypeTag = CreateTypeElement(Type['ServiceType']) - for Service in Type['ServiceList']: - ServiceTag = CreateServiceElement(Service['ServiceId'], Service['GroupId'], Service['Status']) - TypeTag.append(ServiceTag) - root.append(TypeTag) - return root - - -def CreateTypeElement(ServiceType): - NewTypeTag = ET.Element("SERVICETYPE") - # last_updated의 text를 지정한다 - # NewDiskTag.text = '\n\t' - NewTypeTag.attrib["type"] = ServiceType - return NewTypeTag - - -def CreateServiceElement(ServiceId, GroupId, Status): - NewServiceTag = ET.Element("SERVICE") - # last_updated의 text를 지정한다 - # NewDiskTag.text = '\n\t' - NewServiceTag.attrib["id"] = ServiceId - NewServiceTag.attrib["group"] = GroupId - NewServiceTag.attrib["status"] = Status - return NewServiceTag - - -def indent(elem, level=0): - i = "\n" + level*" " - if len(elem): - if not elem.text or not elem.text.strip(): - elem.text = i + " " - if not elem.tail or not elem.tail.strip(): - elem.tail = i - for elem in elem: - indent(elem, level+1) - if not elem.tail or not elem.tail.strip(): - elem.tail = i - else: - if level and (not elem.tail or not elem.tail.strip()): - elem.tail = i - diff --git a/core/common/setup.py b/core/common/setup.py index 6cd58165..60e61ef3 100644 --- a/core/common/setup.py +++ b/core/common/setup.py @@ -16,7 +16,7 @@ ('/usr/local/ksan/sbin/', ['./ksanAgent']), ('/var/log/ksan/rabbitmq', []), ('/var/log/ksan/agent', []) ],\ - packages=['server', 'disk','common', 'configure', 'const' \ + packages=['portal_api','server', 'disk','common', 'configure', 'const' \ ,'Enums','mqmanage','network', \ 'server','service','user', 'util', 'rest'], ) diff --git a/core/common/start.sh b/core/common/start.sh deleted file mode 100644 index 4321e04b..00000000 --- a/core/common/start.sh +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/bash -python /usr/local/ksan/bin/ksanEdge start -python /usr/local/ksan/bin/ksanMon start \ No newline at end of file diff --git a/core/common/user/user_manage.py b/core/common/user/user_manage.py index b9cf2515..00c9d24c 100644 --- a/core/common/user/user_manage.py +++ b/core/common/user/user_manage.py @@ -13,11 +13,12 @@ import os import sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from server.server_manage import * +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from const.common import * from disk.diskpool_manage import GetDefaultDiskPool -from const.http import ResponseHeaderModule -from const.user import AddUserObject, S3UserObjectModule, S3UserStorageClassObject, S3UserObject, S3UserUpdateObject +from const.user import AddUserObject, S3UserStorageClassObject, S3UserObject, S3UserUpdateObject +from common.utils import * @catch_exceptions() @@ -273,7 +274,6 @@ def RemoveS3UserStorageClass(ip, port, ApiKey, StorageClass, UserId=None, UserNa return Res, Errmsg, Ret - @catch_exceptions() def ShowS3UserInfo(UserList, Detail=None, SysinfoDisp=False): """ @@ -282,27 +282,28 @@ def ShowS3UserInfo(UserList, Detail=None, SysinfoDisp=False): :param NicId: :return: """ + UserList = UserListOrdering(UserList) if SysinfoDisp is True: UserTitleLine = '=' * 105 UserDataLine = '-' * 105 - title = "|%s|%s|%s|" % ('User'.center(25), 'AccessKey'.center(32), 'SecretKey'.center(44)) + title = "|%s|%s|%s|" % ('UserName'.ljust(25), 'AccessKey'.ljust(32), 'SecretKey'.ljust(44)) UserDiskPoolStorageClassLine = '' UserDiskPoolStorageClassTitle = '' else: if Detail is None: UserTitleLine = '=' * 92 UserDataLine = '-' * 92 - title ="|%s|%s|%s|" % ('Name'.center(20), 'Email'.center(30), 'Id'.center(38)) + title ="|%s|%s|%s|" % ('UserName'.ljust(20), 'Email'.ljust(30), 'UserId'.ljust(38)) UserDiskPoolStorageClassLine = '' UserDiskPoolStorageClassTitle = '' else: UserTitleLine = '=' * 166 UserDataLine = '-' * 166 - title = "|%s|%s|%s|%s|%s|" % ('Name'.center(20), 'AccessKey'.center(30), - 'SecretKey'.center(42), 'Email'.center(30), 'Id'.center(38)) + title = "|%s|%s|%s|%s|%s|" % ('UserName'.ljust(20), 'AccessKey'.ljust(30), + 'SecretKey'.ljust(42), 'Email'.ljust(30), 'UserId'.ljust(38)) UserDiskPoolStorageClassLine = '%s%s' % (' ' * 101, '-' * 65) - UserDiskPoolStorageClassTitle = "%s|%s|%s|%s|" % (' ' * 101, "DiskPoolName".center(20), "StorageClass".center(20), " "*21) + UserDiskPoolStorageClassTitle = "%s|%s|%s|%s|" % (' ' * 101, "DiskPoolName".ljust(20), "StorageClass".ljust(20), " "*21) print(UserTitleLine) print(title) @@ -313,15 +314,15 @@ def ShowS3UserInfo(UserList, Detail=None, SysinfoDisp=False): user.Email = user.Email if user.Email is not None else '' if SysinfoDisp is True: - _user = "|%s|%s|%s|" % ('{:25.25}'.format(str(user.Name).center(25)), user.AccessKey.center(32), user.SecretKey.center(44)) + _user = "|%s|%s|%s|" % ('{:25.25}'.format(str(user.Name).ljust(25)), user.AccessKey.ljust(32), user.SecretKey.ljust(44)) else: if Detail is None: - _user ="|%s|%s|%s|" % ('{:20.20}'.format(str(user.Name).center(20)), user.Email.center(30), user.Id.center(38)) + _user ="|%s|%s|%s|" % ('{:20.20}'.format(str(user.Name).ljust(20)), user.Email.ljust(30), user.Id.ljust(38)) else: - _user ="|%s|%s|%s|%s|%s|" % ('{:20.20}'.format(str(user.Name).center(20)), - user.AccessKey.center(30), user.SecretKey.center(42), user.Email.center(30), user.Id.center(38)) + _user ="|%s|%s|%s|%s|%s|" % ('{:20.20}'.format(str(user.Name).ljust(20)), + user.AccessKey.ljust(30), user.SecretKey.ljust(42), user.Email.ljust(30), user.Id.ljust(38)) for diskpool in user.UserDiskPools: - _userdiskpool += "%s|%s|%s|%s|\n" % (' ' * 101, diskpool['DiskPoolName'].center(20), diskpool['StorageClass'].center(20), " "*21) + _userdiskpool += "%s|%s|%s|%s|\n" % (' ' * 101, diskpool['DiskPoolName'].ljust(20), diskpool['StorageClass'].ljust(20), " "*21) _userdiskpool += "%s%s\n" % (' ' * 101, '-' * 65) print(_user) @@ -332,10 +333,23 @@ def ShowS3UserInfo(UserList, Detail=None, SysinfoDisp=False): print(_userdiskpool) print(UserDataLine) else: - print('No user data'.center(105)) + print('No user data'.ljust(105)) print(UserDataLine) +def UserListOrdering(UserList): + NewUserList = list() + UserNameDict = dict() + for userinfo in UserList: + UserName = userinfo.Name + UserNameDict[UserName] = userinfo + + for username in sorted(UserNameDict.keys(), key=str.casefold): + userinfo = UserNameDict[username] + NewUserList.append(userinfo) + + return NewUserList + def UserUtilHandler(Conf, Action, Parser, logger): options, args = Parser.parse_args() @@ -369,7 +383,7 @@ def UserUtilHandler(Conf, Action, Parser, logger): Res, Errmsg, Ret = AddS3User(PortalIp, PortalPort, PortalApiKey, options.UserName, DiskPoolName=DefaultDiskpoolName, Email=options.Email, logger=logger) if Res == ResOk: - print(Ret.Result) + print(Ret.Result, Ret.Message) else: print(Errmsg) elif Action.lower() == 'remove': @@ -382,7 +396,7 @@ def UserUtilHandler(Conf, Action, Parser, logger): if Conf['isRemove'] == 'yes': Res, Errmsg, Ret = RemoveS3User(PortalIp, PortalPort, PortalApiKey, UserName=options.UserName, logger=logger) if Res == ResOk: - print(Ret.Result) + print(Ret.Result, Ret.Message) else: print(Errmsg) @@ -391,10 +405,14 @@ def UserUtilHandler(Conf, Action, Parser, logger): Parser.print_help() sys.exit(-1) + if options.StorageClass not in ValidStorageClassList: + print('Error : Unsupported User-defined Storage Class Name - Please Insert s3-compatible storage class (eg. glacier)') + sys.exit(-1) + Res, Errmsg, Ret = AddS3UserStorageClass(PortalIp, PortalPort, PortalApiKey ,options.StorageClass, UserName=options.UserName, DiskPoolName=options.DiskpoolName, logger=logger) if Res == ResOk: - print(Ret.Result) + print(Ret.Result, Ret.Message) else: print(Errmsg) @@ -403,10 +421,14 @@ def UserUtilHandler(Conf, Action, Parser, logger): Parser.print_help() sys.exit(-1) + if options.StorageClass not in ValidStorageClassList: + print('Error : Unsupported User-defined Storage Class Name - Please Insert s3-compatible storage class (eg. glacier)') + sys.exit(-1) + Res, Errmsg, Ret = RemoveS3UserStorageClass(PortalIp, PortalPort, PortalApiKey, options.StorageClass, UserName=options.UserName, DiskPoolName=options.DiskpoolName, logger=logger) if Res == ResOk: - print(Ret.Result) + print(Ret.Result, Ret.Message) else: print(Errmsg) diff --git a/core/common/util/cbalance.py b/core/common/util/cbalance.py index 8e0f2da6..df9ea276 100644 --- a/core/common/util/cbalance.py +++ b/core/common/util/cbalance.py @@ -13,12 +13,11 @@ import os import sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.shcommand import shcall -from server.server_manage import * -from disk.diskpool_manage import GetDefaultDiskPool -from const.http import ResponseHeaderModule -from const.user import AddUserObject, S3UserObjectModule, S3UserStorageClassObject, S3UserObject, S3UserUpdateObject +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +#from server.server_api import * +from const.common import * +from common.utils import * def CbalanceUtilHandler(Conf, Action, Parser, logger): @@ -39,7 +38,7 @@ def CbalanceUtilHandler(Conf, Action, Parser, logger): if not options.BucketName: #print('Bucket Name is required') isValid = False - elif not (options.Key or options.ObjectId): + elif not (options.Key or options.ObjId): #print('Key or ObjectId is required') isValid = False elif not (options.DstDiskName or options.VersionId or options.SrcDiskName): @@ -49,15 +48,15 @@ def CbalanceUtilHandler(Conf, Action, Parser, logger): Parser.print_help() sys.exit(-1) - CbalanceCmd = 'java -jar %s/%s ' % (KsanUtilDirPath, TypeServiceCbalance) + CbalanceCmd = 'java -jar %s/%s.jar ' % (KsanUtilDirPath, TypeServiceCbalance) CbalanceCmd += '--BucketName %s' % options.BucketName if options.Key: CbalanceCmd += ' --Key %s' % options.Key else: - CbalanceCmd += ' --ObjectId %s' % options.ObjectId + CbalanceCmd += ' --ObjId %s' % options.ObjId if options.VersionId: - CbalanceCmd += ' --VersionId ' % options.VersionId + CbalanceCmd += ' --VersionId %s' % options.VersionId if options.DstDiskName: CbalanceCmd += ' --DstDiskName %s' % options.DstDiskName if options.SrcDiskName: diff --git a/core/common/util/fsck.py b/core/common/util/fsck.py index 746df521..48d9793f 100644 --- a/core/common/util/fsck.py +++ b/core/common/util/fsck.py @@ -13,12 +13,10 @@ import os import sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.shcommand import shcall -from server.server_manage import * -from disk.diskpool_manage import GetDefaultDiskPool -from const.http import ResponseHeaderModule -from const.user import AddUserObject, S3UserObjectModule, S3UserStorageClassObject, S3UserObject, S3UserUpdateObject +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from const.common import * +from common.utils import * def FsckUtilHandler(Conf, Action, Parser, logger): @@ -39,7 +37,7 @@ def FsckUtilHandler(Conf, Action, Parser, logger): Parser.print_help() sys.exit(-1) - FsckCmd = 'java -jar %s/%s ' % (KsanUtilDirPath, TypeServiceFsck) + FsckCmd = 'java -jar %s/%s.jar ' % (KsanUtilDirPath, TypeServiceFsck) if options.BucketName: FsckCmd += '--BucketName %s' % options.BucketName else: diff --git a/core/common/util/getattr.py b/core/common/util/getattr.py index 817367fc..0810f170 100644 --- a/core/common/util/getattr.py +++ b/core/common/util/getattr.py @@ -13,12 +13,10 @@ import os import sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.shcommand import shcall -from server.server_manage import * -from disk.diskpool_manage import GetDefaultDiskPool -from const.http import ResponseHeaderModule -from const.user import AddUserObject, S3UserObjectModule, S3UserStorageClassObject, S3UserObject, S3UserUpdateObject +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from const.common import * +from common.utils import * def GetAttrUtilHandler(Conf, Action, Parser, logger): @@ -35,15 +33,22 @@ def GetAttrUtilHandler(Conf, Action, Parser, logger): sys.exit(-1) if Action.lower() == 'getattr': - if not (options.BucketName or options.Key): + if len(args) != 2: Parser.print_help() + print('bucket name is required') sys.exit(-1) - GetAttrCmd = 'java -jar %s/%s ' % (KsanUtilDirPath, TypeServiceGetAttr) - GetAttrCmd += '--BucketName %s' % options.BucketName + BucketName = args[1] + + if not (options.ObjectId or options.Key): + Parser.print_help() + sys.exit(-1) + + GetAttrCmd = 'java -jar %s/%s.jar ' % (KsanUtilDirPath, TypeServiceGetAttr) + GetAttrCmd += '--BucketName %s' % BucketName if options.Key: - GetAttrCmd += ' --Key %s' % options.Key + GetAttrCmd += ' --Key "%s"' % options.Key else: GetAttrCmd += ' --ObjectId %s' % options.ObjectId if options.VersionId: @@ -51,6 +56,7 @@ def GetAttrUtilHandler(Conf, Action, Parser, logger): out, err = shcall(GetAttrCmd) print(out, err) + logger.debug("%s %s %s" % (GetAttrCmd, out, err)) else: Parser.print_help() diff --git a/core/common/util/sysinfo.py b/core/common/util/sysinfo.py index e1202f79..b1725096 100644 --- a/core/common/util/sysinfo.py +++ b/core/common/util/sysinfo.py @@ -12,10 +12,10 @@ import os, sys import psutil -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from server.server_manage import GetAllServerDetailInfo, ShowServerInfo -from service.service_manage import ShowServiceInfoWithServerInfo, ShowServiceInfo -from disk.diskpool_manage import GetDiskPoolInfo, ShowDiskPoolInfo +if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: + sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) +from portal_api.apis import * +from disk.diskpool_manage import GetDiskPoolInfo, ShowDiskPoolInfoNew from user.user_manage import GetS3UserInfo, ShowS3UserInfo from const.common import ResOk, ResultSuccess, MoreDetailInfo, SimpleInfo, DetailInfo @@ -39,7 +39,8 @@ def ShowSystemInfo(PortalIp, PortalPort, PortalApiKey, logger, Detail=False): # Get Service Info print("\n[KSAN Service Information]") - Res, Errmsg, Ret, Data = GetAllServerDetailInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) + #Res, Errmsg, Ret, Data = GetAllServerDetailInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) + Res, Errmsg, Ret, Data = GetServiceInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) if Res == ResOk: if Ret.Result == ResultSuccess: @@ -55,18 +56,8 @@ def ShowSystemInfo(PortalIp, PortalPort, PortalApiKey, logger, Detail=False): if Res != ResOk: print(Errmsg) else: - if DispLevel is True: - Detail = MoreDetailInfo - elif DispLevel: - Detail = DetailInfo - else: - Detail = SimpleInfo - - Res, Errmsg, Ret, ServerDetailInfo = GetAllServerDetailInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - ShowDiskPoolInfo(DiskPoolList, ServerDetailInfo, Detail=True, SysinfoDisp=SysinfoDisp) + Detail = SimpleInfo + ShowDiskPoolInfoNew(DiskPoolList, Detail=Detail, SysinfoDsp=True) print("\n[KSAN User Information]") Res, Errmsg, Ret, Users = GetS3UserInfo(PortalIp, PortalPort, PortalApiKey, logger=logger) diff --git a/core/mgs/util/ksan b/core/mgs/util/ksan index 28085ad8..a42d9c0a 100755 --- a/core/mgs/util/ksan +++ b/core/mgs/util/ksan @@ -12,7 +12,8 @@ import os, sys sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from disk.disk_manage import DiskUtilHandler +sys.path.insert(1, os.getcwd()) +from disk.disk_api import DiskUtilHandler from service.service_manage import ServiceUtilHandler from configure.config import ConfigUtilHandler from disk.diskpool_manage import DiskpoolUtilHandler @@ -22,9 +23,7 @@ from util.fsck import FsckUtilHandler from util.getattr import GetAttrUtilHandler from util.cbalance import CbalanceUtilHandler from util.sysinfo import ShowSystemInfo -from network.network_manage import * -from common.init import * -from common.utils import AvoidSigInt +from common.base_utils import * class KsanOptionParser(OptionParser): @@ -33,10 +32,10 @@ class KsanOptionParser(OptionParser): self.Usage = """ ksan [sysinfo]|[server|service|disk|diskpool|user] [actions] [option]* [actions] - sysinfo : display total system info - server [add|remove|list] : server management + sysinfo : display summary of system info + server [add|remove|list|update] : server management service [add|remove|list|start|stop|restart] : service management - config [add|remove|list|set|get] : serivce config management + config [add|remove|list|set|get] : service config management disk [add|remove|list|start|stop|set] : disk management diskpool [add|remove|list|add2disk|remove2disk|set] : diskpool management user [add|remove|list|add2storageclass|remove2storageclass|set] : ksan user management @@ -48,18 +47,33 @@ class KsanOptionParser(OptionParser): --ServerName : server name --ServiceName : service name --DiskName : disk name - --DiskpoolName : diskpool name + --DiskPoolName : diskpool name --UserName : user name - --ServiceType : service type [ksanAgent|ksanOSD|ksanGW|ksanRecovery| - ksanLifecycle|ksanReplication|ksanMetering] - --ConfVerId : config version - --RepType : replication type[1|2] 1: 1+0, 2:1+1 - --DiskpoolType : diskpool type[standard|archive] - --DefaultDiskpool : set user's default diskpool - --StorageClass : user's storage class[standard_ia|onezone_ia|intelligent_tiering| - glacier|reduced_redundancy|deep_archive|outposts|glacier_ir] + --ServiceType : service type [ksanAgent|ksanOSD|ksanGW| + ksanLifecycleManager|ksanReplicationManager|ksanLogManager] + --ConfVerId : specific config version id + 'last' value specify the latest config version id. + --Tolerance : diskpool's fault tolerance [disable|replication|ec(k:m)] + ec(k:m): Erasure Code (the number of data chunks : the number of coding chunks) + --DiskPoolType : diskpool type[standard|archive] + --DefaultDiskPool : set user's default diskpool + --StorageClass : user-defined storage class for s3-compatible --Host : server ip or hostname - --DiskPath : osd disk mount path + --DiskPath : osd disk's mount path + + --BucketName : specify the name of the bucket + --KeyName : specify the object key + --ObjId : specify the object Id instead of object key + --SrcDiskName : specify the source disk Name + --DstDiskName : specify the destination disk name + --EmptyDisk : set to enable empty disk operation + --OkLocalMove : allow to move to local another disk (default: false) + --Size : specify the capacity to move in KB/KiB, MB/MiB, GB/GiB, TB/TiB, and PB/PiB units + --CheckOnly : specify if you wish only to check not to fix + --VersionId : specify the object version Id if you wish particular version of an object + --Checksum : display the checksum and size of the object from ksanOSD + + [option] --v : display info in detail @@ -78,12 +92,12 @@ class KsanOptionParser(OptionParser): self.add_option("--ConfVerId", "--confverid", dest="ConfigVersionId", help='service config version id') self.add_option("--DiskName", "--diskname", dest="DiskName", help='disk name') - self.add_option("--DiskPath", "--diskpath", dest="DiskPath", help='osd disk mount path') - self.add_option("--DiskpoolName", "--diskpoolname", dest="DiskpoolName", help='diskpool name') - self.add_option("--RepType", "--reptype", dest="RepType", help='replication type') - self.add_option("--DiskpoolType", "--diskpooltype",dest="DiskpoolType", help='diskpool type') + self.add_option("--DiskPath", "--diskpath", dest="DiskPath", help='osd disk\'s mount path') + self.add_option("--DiskPoolName", "--DiskpoolName", "--diskpoolname", dest="DiskpoolName", help='diskpool name') + self.add_option("--Tolerance", "--tolerance", dest="RepType", help='replication type') + self.add_option("--DiskPoolType", "--DiskpoolType", "--diskpooltype",dest="DiskpoolType", help='diskpool type') self.add_option("--UserName", "--username", dest="UserName", help='user name') - self.add_option("--DefaultDiskpool", "--defaultdiskpool", dest="DefaultDiskpool", help='user\'s default diskpool') + self.add_option("--DefaultDiskPool", "--DefaultDiskpool", "--defaultdiskpool", dest="DefaultDiskpool", help='user\'s default diskpool') self.add_option("--ConfFile", "--conffile", dest="ConfFile", help='config file path') self.add_option("--Email", "--email", dest="Email", help='user\' e-mail address') self.add_option("--V", "--v", dest="Detail", action="store_true", help='display information in detail') @@ -123,10 +137,12 @@ class KsanServerUtilOptParser(OptionParser): add : register server remove : remove server list : show server info + update : update server info [options] --ServerName : server name --Host : target server ip or hostname --v : show info in detail + --vv : show info in more detail [example] 1. register server @@ -139,12 +155,14 @@ class KsanServerUtilOptParser(OptionParser): 3. show server info # ./ksan server list # ./ksan server list --v + # ./ksan server list --vv """ self.AddOption() def AddOption(self): self.add_option('-d', "--Debug", dest='debug', action='store_true') self.add_option("--ServerName", "--servername", dest="ServerName", help='server name') + self.add_option("--NetworkName", "--networkname", dest="NetworkName", help='network name') self.add_option("--Host", "--host", dest="Host", help='server ip or hostname') self.add_option("--V", "--v", dest="Detail", action="store_true", help='display information in detail') self.add_option("--VV", "--vv", dest="MoreDetail", action="store_true", help='display information in more detail') @@ -167,25 +185,27 @@ class KsanServiceUtilOptParser(OptionParser): [options] --ServerName : server name --ServiceName : service name - --ServiceType : service type [ksanAgent|ksanOSD|ksanGW|ksanRecovery| - ksanLifecycle|ksanReplication|ksanMetering] - --V : show info in detail + --ServiceType : service type [ksanAgent|ksanOSD|ksanGW| + ksanLifecycleManager|ksanReplicationManager|ksanLogManager] + --v : show info in detail + --vv : show info in more detail [example] 1. register service - # ./ksan service add --ServiceType ksanosd --ServiceName ksan_osd1 --ServerName oosd1 - # ./ksan service add --ServiceType ksangw --ServiceName ksan_gw2 --ServerName oosd2 + # ./ksan service add --ServiceType ksanosd --ServiceName osd1 --ServerName server1 + # ./ksan service add --ServiceType ksangw --ServiceName gw2 --ServerName server2 2. remove service - # ./ksan service add --ServiceName ksan_osd1 + # ./ksan service remove --ServiceName osd1 3. show service info # ./ksan service list # ./ksan service list --v + # ./ksan service list --vv - 4. start/stop/restart servcie - # ./ksan service start --ServiceName ksan_osd1 - # ./ksan service stop --ServiceName ksan_osd1 + 4. start/stop/restart service + # ./ksan service start --ServiceName osd1 + # ./ksan service stop --ServiceName osd1 """ self.AddOption() @@ -195,6 +215,7 @@ class KsanServiceUtilOptParser(OptionParser): self.add_option("--ServiceName", "--servicename",dest="ServiceName", help='service name') self.add_option("--ServiceType", "--servicetype", dest="ServiceType", help='service type') self.add_option("--V", "--v", dest="Detail", action="store_true", help='display information in detail') + self.add_option("--VV", "--vv", dest="MoreDetail", action="store_true", help='display information in detail') def print_help(self): print(self.Usage) @@ -210,28 +231,33 @@ class KsanConfigUtilOptParser(OptionParser): set : apply config get : show the current applied config [options] - --ServiceType : service type [ksanAgent|ksanOSD|ksanGW|ksanRecovery| - ksanLifecycle|ksanReplication|ksanMetering] - --ConfVerId : config version id + --ServiceType : service type [ksanAgent|ksanOSD|ksanGW| + ksanLifecycleManager|ksanReplicationManager|ksanLogManager] + --ConfVerId : specific config version id + 'last' value scpecifies the latest config version id + --ConfFile : save current config to file or add config from file [example] 1. add config # ./ksan config add --ServiceType ksanosd + # ./ksan config add --ServiceType ksanosd --ConfFile ksanosd.conf 2. remove config - # ./ksan config remove --ConfVerId 5 --ServiceType ksangw + # ./ksan config remove --ServiceType ksangw --ConfVerId 5 3. show config list # ./ksan config list --ServiceType ksanosd # ./ksan config list --ServiceType ksangw 4. set config # ./ksan config set --ConfVerId 5 --ServiceType ksangw + # ./ksan config set --ConfVerId last --ServiceType ksangw 5. get config # ./ksan config get --ServiceType ksanosd + # ./ksan config get --ServiceType ksanosd --ConfFile ksanosd.conf """ self.AddOption() def AddOption(self): self.add_option('-d', "--Debug", dest='debug', action='store_true') self.add_option("--ServiceType" , "--servicetype", dest="ServiceType", help='service type') - self.add_option("--ConfVerId", "--confverid", dest="ConfigVersionId", help='config version id') - self.add_option("--ConfFile", "--conffile", dest="ConfFile", help='config file') + self.add_option("--ConfVerId", "--ConfVerid", "--Confverid", "--confverid", dest="ConfigVersionId", help='config version id') + self.add_option("--ConfFile", "--Conffile", "--conffile", dest="ConfFile", help='config file') def print_help(self): print(self.Usage) @@ -254,8 +280,8 @@ class KsanDiskUtilOptParser(OptionParser): --DiskName : disk name --DiskPath : disk mount path --Mode : disk mode [ro|readonly|rw|readwrite] - --V : show info in detail - --VV : show info in more detail + --v : show info in detail + --vv : show info in more detail --Continue : display info in every n seconds [example] @@ -286,7 +312,7 @@ class KsanDiskUtilOptParser(OptionParser): self.add_option('-d', "--Debug", dest='debug', action='store_true') self.add_option("--ServerName", "--servername", dest="ServerName", help='server name') self.add_option("--DiskName", "--diskname", dest="DiskName", help='disk name') - self.add_option("--DiskPath", "--diskpath", dest="DiskPath", help='osd disk mount path') + self.add_option("--DiskPath", "--diskpath", dest="DiskPath", help='osd disk\'s mount path') self.add_option("--Mode", "--mode", dest="Mode", help='disk mode') self.add_option("--V", "--v", dest="Detail", action="store_true", help='display information in detail') self.add_option("--VV", "--vv", dest="MoreDetail", action="store_true", help='display information in more detail') @@ -309,49 +335,48 @@ class KsanDiskpoolUtilOptParser(OptionParser): get : get diskpool default diskpool [options] - --DiskpoolName : diskpool name + --DiskPoolName : diskpool name --DiskName : disk name - --RepType : disk replica type [1|2] 1: 1+0, 2: 1+1 - --DiskpoolType : diskpool type [standard|archive] - --DefaultDiskpool : default diskpool name - --V : show info in detail - --VV : show info in more detail - --Continue : display info in every n seconds + --Tolerance : diskpool's fault tolerance [disable|replication|ec(k:m)] + ec(k:m): Erasure Code (the number of data chunks : the number of coding chunks) + --DiskPoolType : diskpool type [standard|archive] + --DefaultDiskPool : system's default diskpool + --v : show info in detail + --Continue : show info in every n seconds [example] 1. add diskpool - # ./ksan diskpool add --DiskpoolName diskpool1 --RepType 2 --DiskpoolType standard + # ./ksan diskpool add --DiskPoolName diskpool1 --Tolerance replication --DiskPoolType standard + # ./ksan diskpool add --DiskPoolName diskpool1 --Tolerance ec(6:2) --DiskPoolType archive 2. remove diskpool - # ./ksan diskpool remove --DiskpoolName diskpool1 + # ./ksan diskpool remove --DiskPoolName diskpool1 3. show diskpool list # ./ksan diskpool list # ./ksan diskpool list --v - # ./ksan diskpool list --vv # ./ksan diskpool list --v --Continue 1 4. add disk to diskpool - # ./ksan diskpool add2disk --DiskpoolName diskpool1 --DiskName disk1 - # ./ksan diskpool remove2disk --DiskpoolName diskpool1 --DiskName disk1 + # ./ksan diskpool add2disk --DiskPoolName diskpool1 --DiskName disk1 + # ./ksan diskpool remove2disk --DiskPoolName diskpool1 --DiskName disk1 5. set/get diskpool default diskpool - # ./ksan diskpool set --DefaultDiskpool --DiskpoolName diskpool1 - # ./ksan diskpool get --DefaultDiskpool + # ./ksan diskpool set --DefaultDiskPool --DiskPoolName diskpool1 + # ./ksan diskpool get --DefaultDiskPool """ self.AddOption() def AddOption(self): self.add_option('-d', "--Debug", "--debug", dest='debug', action='store_true') - self.add_option("--DiskpoolName", "--diskpoolname", dest="DiskpoolName", help='diskpool name') + self.add_option("--DiskpoolName", "--DiskPoolName","--diskpoolname", dest="DiskpoolName", help='diskpool name') self.add_option("--DiskName", "--diskname", dest="DiskName", help='disk name') - self.add_option("--RepType", "--reptype", dest="RepType", help='replication type') - self.add_option("--DiskpoolType", "--diskpooltype", dest="DiskpoolType", help='diskpool type') - self.add_option("--DefaultDiskpool", "--defaultdiskpool", dest="DefaultDiskpool", action="store_true", help='user\'s default diskpool') - self.add_option("--V", "--v", dest="Detail", action="store_true", help='display information in detail') - self.add_option("--VV", "--vv", dest="MoreDetail", action="store_true", help='display information in more simple') - self.add_option("--Continue", "--continue", dest="Continue", type="int", help='display information continually in every n seconds') + self.add_option("--Tolerance", "--tolerance", dest="RepType", help='replication type') + self.add_option("--DiskpoolType", "--Diskpooltype","--DiskPoolType", "--diskpooltype", dest="DiskpoolType", help='diskpool type') + self.add_option("--DefaultDiskPool", "--DefaultdiskPool", "--defaultdiskpool", "--DefaultDiskpool", dest="DefaultDiskpool", action="store_true", help='user\'s default diskpool') + self.add_option("--V", "--v", dest="Detail", action="store_true", help='show information in detail') + self.add_option("--Continue", "--continue", dest="Continue", type="int", help='show information continually in every n seconds') def print_help(self): print(self.Usage) @@ -364,22 +389,21 @@ class KsanUserUtilOptParser(OptionParser): add : add user remove : remove user list : show user info - add2storagegclass : add storage class to user - remove2storagegclass : remove storage slass from user + add2storageclass : add storage class to user + remove2storageclass : remove storage class from user set : change user email address [options] --UserName : user name - --DiskpoolName : diskpool name - --DefaultDiskpool : user's default diskpool - --StorageClass : user's storage class[standard_ia|onezone_ia|intelligent_tiering| - glacier|reduced_redundancy|deep_archive|outposts|glacier_ir] + --DiskPoolName : diskpool name + --DefaultDiskPool : user's default diskpool + --StorageClass : user-defined storage class for s3-compatible --Email : user's Email - --V : show info in detail + --v : show info in detail [example] 1. add user - # ./ksan user add --UserName user1 --Email user1@pspace.co.kr --DefaultDiskpool diskpool1 + # ./ksan user add --UserName user1 --Email user1@pspace.co.kr --DefaultDiskPool diskpool1 # ./ksan user add --UserName user1 --Email user1@pspace.co.kr 2. remove user @@ -390,8 +414,8 @@ class KsanUserUtilOptParser(OptionParser): # ./ksan user list --v 4. add/remove storage class to user - # ./ksan user add2storageclass --UserName user1 --DiskpoolName diskpool1 --StorageClass glacier - # ./ksan user remove2storageclass --UserName user1 --DiskpoolName diskpool1 --StorageClass glacier + # ./ksan user add2storageclass --UserName user1 --DiskPoolName diskpool1 --StorageClass classname + # ./ksan user remove2storageclass --UserName user1 --DiskPoolName diskpool1 --StorageClass classname 5. change user's email address # ./ksan user set --UserName user1 --Email user1@pspace.com @@ -402,8 +426,8 @@ class KsanUserUtilOptParser(OptionParser): self.add_option('-d', "--Debug", dest='debug', action='store_true') self.add_option("--UserId", "--userid", dest="UserId", help='user id') self.add_option("--UserName", "--username", dest="UserName", help='user name') - self.add_option("--DiskpoolName", "--diskpoolname", dest="DiskpoolName", help='diskpool name') - self.add_option("--DefaultDiskpool", "--defaultdiskpool", dest="DefaultDiskpool", help='user\'s default diskpool') + self.add_option("--DiskpoolName", "--DiskPoolName","--diskpoolname", dest="DiskpoolName", help='diskpool name') + self.add_option("--DefaultDiskpool", "--DefaultDiskPool", "--defaultdiskpool", dest="DefaultDiskpool", help='user\'s default diskpool') self.add_option("--StorageClass", "--storageclass", dest="StorageClass", help='user\'s storage class') self.add_option("--Email", "--email", dest="Email", help='user\' e-mail address') self.add_option("--V", "--v", dest="Detail", action="store_true", help='display information in detail') @@ -434,44 +458,44 @@ class KsanFsckUtilOptParser(OptionParser): self.AddOption() def AddOption(self): - self.add_option("--BucketName", "--bucketname", dest="BucketName", help='bucket name') - self.add_option("--DiskName", "--DiskName", dest="DiskName", help='disk name') - self.add_option("--CheckOnly", "--checkonly", dest="CheckOnly", action='store_true', help='check only mode') + self.add_option("--BucketName", "--Bucketname", "--bucketname", dest="BucketName", help='bucket name') + self.add_option("--DiskName", "--Diskname", "--diskname", dest="DiskName", help='disk name') + self.add_option("--CheckOnly", "--Checkonly", "--checkonly", dest="CheckOnly", action='store_true', help='check only mode') self.add_option('--Debug', "--debug", dest='debug', action='store_true', help='display debug log') def print_help(self): print(self.Usage) + class KsanGetAttrUtilOptParser(OptionParser): def __init__(self): super(KsanGetAttrUtilOptParser, self).__init__() self.Usage = """ - ksan getattr [option]* + ksan getattr [BucketName] [option]* [options] - --BucketName : specify the name of the bucket --Checksum : display the checksum and size of the object from ksanOSD - --Key : specify the object key - --ObjectId : specify the object Id if you wish to display with Id rather than object key + --KeyName : specify the object key + --ObjId : specify the object Id if you with to display with Id rather than object key --VersionId : specify the object version Id if you wish particular version of an object [example] - ksan getattr --BucketName bucket1 --Key file1.txt - ksan getattr --BucketName bucket1 --Key file1.txt --Checksum - ksan getattr --BucketName bucket1 --Key file1.txt --VersionId 526554498818254 - ksan getattr --BucketName bucket1 --ObjId bd01856bfd2065d0d1ee20c03bd3a9af - ksan getattr --BucketName bucket1 --ObjId bd01856bfd2065d0d1ee20c03bd3a9af --VersionId 526554498818254 + ksan getattr bucket1 --KeyName file1.txt + ksan getattr bucket1 --KeyName file1.txt --Checksum + ksan getattr bucket1 --KeyName file1.txt --VersionId 526554498818254 + ksan getattr bucket1 --ObjId bd01856bfd2065d0d1ee20c03bd3a9af + ksan getattr bucket1 --ObjId bd01856bfd2065d0d1ee20c03bd3a9af --VersionId 526554498818254 """ self.AddOption() def AddOption(self): - self.add_option("--BucketName", "--bucketname", dest="BucketName", help='bucket name') - self.add_option("--Checksum", "--checksum", dest="Checksum", help='display checksum and size of the object from ksanOSD') - self.add_option("--Key", "--key", dest="Key", help='object key') - self.add_option("--ObjectId", "--objectid", dest="ObjectId", help='object id') - self.add_option("--VersionId", "--versionid", dest="VersionId", help='version id') + #self.add_option("--BucketName", "--bucketname", dest="BucketName", help='bucket name') + self.add_option("--Checksum", "--checksum", action='store_true', dest="Checksum", help='display checksum and size of the object from ksanOSD') + self.add_option("--KeyName", "--Keyname", "--keyname", dest="Key", help='object key') + self.add_option("--ObjId", "--Objid", "--objid", dest="ObjectId", help='object id') + self.add_option("--VersionId", "--Versionid", "--versionid", dest="VersionId", help='version id') self.add_option('--Debug', "--debug", dest='debug', action='store_true', help='display debug log') def print_help(self): @@ -485,31 +509,26 @@ class KsanCbalanceUtilOptParser(OptionParser): ksan cbalance [option]* [options] - --BucketName : specify the name of the bucket - --Checksum : display the checksum and size of the object from ksanOSD - --Key : specify the object key - --ObjectId : specify the object Id if you wish to display with Id rather than an object key - --VersionId : specify the object version Id if you wish particular version of an object - --BucketName : specify the name of the bucket you wish to balance - --DstDiskName : specify the destination disk Name + --BucketName : specify the name of the bucket you wish to balance + --KeyName : specify the object key + --DstDiskName : specify the destination disk name --EmptyDisk : set to enable empty disk operation - --Key : specify the object key --ObjId : specify the object Id instead of object key --OkLocalMove : allow to move to local another disk (default: false) --Size : specify the capacity to move in KB/KiB, MB/MiB, GB/GiB, TB/TiB, and PB/PiB units --SrcDiskName : specify the source disk Name --VersionId : specify the object version Id if you wish particular version of an object - --Debug : display debug log to the terminal + --Debug : display debug log to the terminal (default: false) [example] 1. move a single object and the object can be idetified either key or object Id - ksan cbalance --BucketName bucket1 --Key file1.txt --SrcDiskName osd1_disk1 - ksan cbalance --BucketName bucket1 --Key file1.txt --VersionId 526554498818254 --SrcDiskName osd1_disk1 + ksan cbalance --BucketName bucket1 --KeyName file1.txt --SrcDiskName osd1_disk1 + ksan cbalance --BucketName bucket1 --KeyName file1.txt --VersionId 526554498818254 --SrcDiskName osd1_disk1 ksan cbalance --BucketName bucket1 --ObjId bd01856bfd2065d0d1ee20c03bd3a9af --VersionId 526554498818254 --SrcDiskName osd1_disk1 ksan cbalance --BucketName bucket1 --ObjId bd01856bfd2065d0d1ee20c03bd3a9af --SrcDiskName osd1_disk1 - ksan cbalance --BucketName bucket1 --Key file1.txt --DstDiskName osd2_disk2 - ksan cbalance --BucketName bucket1 --Key file1.txt --VersionId 526554498818254 --DstDiskName osd3_disk2 + ksan cbalance --BucketName bucket1 --KeyName file1.txt --DstDiskName osd2_disk2 + ksan cbalance --BucketName bucket1 --KeyName file1.txt --VersionId 526554498818254 --DstDiskName osd3_disk2 2. move a spefic amount of object from one disk to others ksan cbalance --SrcDiskName osd1_disk1 --DstDiskName osd2_disk2 --Size 2GB @@ -522,18 +541,16 @@ class KsanCbalanceUtilOptParser(OptionParser): self.AddOption() def AddOption(self): - self.add_option("--BucketName", "--BucketName", dest="BucketName", help='specify the name of thebucket') - self.add_option("--Checksum", "--Checksum", dest="Checksum", help='display the checksum and size of the object from ksanOSD') - self.add_option("--Key", "--key", dest="Key", help='specify the object key') - self.add_option("--ObjectId", "--ObjectId", dest="ObjectId", help='specify the object Id if you wish to display with Id rather than an object key') - self.add_option("--VersionId", "--VersionId", dest="VersionId", help='specify the object version Id if you wish particular version of an object') - self.add_option("--DstDiskName", "--DstDiskName", dest="DstDiskName", help='specify the destination diskName') - self.add_option("--EmptyDisk", "--EmptyDisk", dest="EmptyDisk", help='set to enable empty disk operation') - self.add_option("--ObjId", "--ObjId", dest="ObjId", help='specify the object Id instead of object key') - self.add_option("--OkLocalMove", "--OkLocalMove", dest="OkLocalMove", help='allow to move to local an other disk(default:false)') - self.add_option("--Size", "--Size", dest="Size", help='specify the capacity to move in KB/KiB, MB/MiB, GB/GiB, TB/TiB, and PB/PiB units') - self.add_option("--SrcDiskName", "--SrcDiskName", dest="SrcDiskName", help='specify the source diskName') - self.add_option("--Debug", "--Debug", dest="debug", help='display debug log to the terminal') + self.add_option("--BucketName", "--bucketname", dest="BucketName", help='specify the name of thebucket') + self.add_option("--KeyName", "--keyname", "--Keyname", dest="Key", help='specify the object key') + self.add_option("--VersionId", "--Versionid", "--versionid", dest="VersionId", help='specify the object version Id if you wish particular version of an object') + self.add_option("--DstDiskName", "--Dstdiskname", "--dstdiskname", dest="DstDiskName", help='specify the destination diskName') + self.add_option("--EmptyDisk", "--Emptydisk", "--emptydisk", dest="EmptyDisk", help='set to enable empty disk operation') + self.add_option("--ObjId", "--objid", "--Objid", dest="ObjId", help='specify the object Id instead of object key') + self.add_option("--OkLocalMove", "--oklocalmove", "--OkLocalmove", dest="OkLocalMove", help='allow to move to local an other disk(default:false)') + self.add_option("--Size", "--size", dest="Size", help='specify the capacity to move in KB/KiB, MB/MiB, GB/GiB, TB/TiB, and PB/PiB units') + self.add_option("--SrcDiskName", "--SrcDiskname", "--srcdiskname", dest="SrcDiskName", help='specify the source diskName') + self.add_option("--Debug", "--debug", dest="debug", help='display debug log to the terminal') def print_help(self): print(self.Usage) diff --git a/core/mgs/util/ksanDisk b/core/mgs/util/ksanDisk deleted file mode 100755 index ca48aca9..00000000 --- a/core/mgs/util/ksanDisk +++ /dev/null @@ -1,192 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import os, sys -import pdb - -if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: - sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from mqmanage.mq import * -from disk.disk_manage import * -from optparse import OptionParser -from common.init import read_conf - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanDisk {add|remove|set|start|stop|update|list} [option] - add -n -p -N : Add disk - remove -N : Remove disk - set -N [Rw|Ro] : Set disk mode. Ro:ReadOnly, Rw: ReadWrite - start -N : Start disk - stop -N : Stop disk - update -N -P -c : Update Disk Info - list -l : Display disk info - [options] - -D : Disk Id - -N : Disk Name - -n : Server Name - -p : Disk Mount Path - -c : Description - -P : Disk Pool Id - -l : show Disk List in Detail - -h, --help : show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {add|remove|set|start|stop|update|list} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-D', "--Id", dest="DiskId", help='Disk id') - parser.add_option('-p', "--Path", dest="Path", help='Disk mount path') - parser.add_option('-P', "--DiskPoolId", dest="DiskPoolId", help='Disk Pool Id') - parser.add_option('-N', "--DiskName", dest="DiskName", help='Disk Name') - parser.add_option('-S', "--Serverid", dest="ServerId", help='Server id') - parser.add_option('-z', "--Size", dest="TotalSize", help='Disk Total Size') - parser.add_option('-c', "--Description", dest="Description", help='Description') - parser.add_option('-n', "--ServerName", dest="ServerName", help='Server Name') - parser.add_option('-l', "--Detail", dest="Detail", action='store_true', help='Detail info') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='Debug mode') - - options, args = parser.parse_args() - - if len(args) < 1: - parser.print_help() - sys.exit(-1) - - IfsPortalIp = '127.0.0.1' - IfsPortalPort = 5443 - IfsMqPort = 5672 - IfsPortalKey = '' - ret, conf = read_conf(MonServicedConfPath) - if ret is True: - IfsPortalIp = conf['mgs']['MgsIp'] - IfsPortalPort = int(conf['mgs']['IfsPortalPort']) - IfsMqPort = int(conf['mgs']['MqPort']) - IfsPortalKey = conf['mgs']['IfsPortalKey'] - - - options.IpAddress = IfsPortalIp - options.Port = IfsPortalPort - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - - #ServerId = None - """ - if args[0] in ['remove', 'update', 'start', 'stop', 'set', 'update_size']: - Res, Errmsg, Ret, AllDiskList = GetDiskInfo(IfsPortalIp, IfsPortalPort, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - if len(AllDiskList) == 0: - print('ServerId is not found with DiskId %s' % options.DiskId) - sys.exit(-1) - else: - Res, Errmsg, Svr = GetServerInfoWithDiskId(AllDiskList, options.DiskId) - if Res == ResOk: - ServerId = Svr.Id - else: - print('fail to get Server Id with DiskId: %s' % Errmsg) - sys.exit(-1) - - else: - print('fail to get ServerId with DiskID') - sys.exit(-1) - """ - - if args[0] == 'add': - if not ((options.ServerId or options.ServerName) and options.DiskName) : - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = AddDisk(IfsPortalIp, IfsPortalPort, IfsPortalKey, options.Path, options.DiskName, - ServerId=options.ServerId, ServerName=options.ServerName, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'remove': - if not (options.DiskId or options.DiskName): - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = RemoveDiskInfo(IfsPortalIp, IfsPortalPort, IfsPortalKey, DiskId=options.DiskId, Name=options.DiskName, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'update': - if not options.DiskId: - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = UpdateDiskInfo(IfsPortalIp, IfsPortalPort, IfsPortalKey, options.DiskId, Path=options.Path, - DiskPoolId=options.DiskPoolId, Name=options.Name, - Description=options.Description, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - - elif args[0] == 'update_size': - if not options.DiskId: - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = UpdateDiskSize(IfsPortalIp, IfsPortalPort, IfsPortalKey, options.DiskId, - TotalSize=options.TotalSize, UsedSize=options.TotalSize, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - elif args[0] == 'set': - if len(args) != 2 or not (options.DiskId or options.DiskName): - parser.print_help() - sys.exit(-1) - else: - if args[1] not in ['Ro', 'Rw']: - parser.print_help() - sys.exit(-1) - DiskMode = DiskModeRw if args[1] == 'Rw' else DiskModeRo - Res, Errmsg, Ret = ChangeDiskMode(IfsPortalIp, IfsPortalPort,IfsPortalKey, DiskMode, DiskId=options.DiskId, - Name=options.DiskName, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - elif args[0] == 'start' or args[0] == 'stop': - if not(options.DiskId or options.DiskName) or args[0] not in ['start', 'stop']: - parser.print_help() - sys.exit(-1) - Action = DiskStart if args[0] == 'start' else DiskStop - Res, Errmsg, Ret = StartStopDisk(options.IpAddress, options.Port, IfsPortalKey, Action, DiskId=options.DiskId, Name=options.DiskName, - logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'list': - while True: - Res, Errmsg, Ret, AllDisks = GetDiskInfo(options.IpAddress, options.Port, IfsPortalKey, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - ShowDiskInfo(AllDisks, ServerId=options.ServerId, DiskId=options.DiskId, Detail=options.Detail) - if options.Continue is None: - break - else: - time.sleep(int(options.Continue)) - else: - parser.print_help() diff --git a/core/mgs/util/ksanDiskpool b/core/mgs/util/ksanDiskpool deleted file mode 100755 index 98589894..00000000 --- a/core/mgs/util/ksanDiskpool +++ /dev/null @@ -1,176 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os, sys -import pdb - -if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: - sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import read_conf -from disk.diskpool_manage import * -from disk.disk_manage import * -from optparse import OptionParser - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanDiskpool {add|remove|update|list} [option] - add -N -r -t : Create Disk Pool - remove -N : Delete Disk Pool - addDisk -N -n <"disk name"> : Add Disk to Pool - removeDisk -N -n <"disk name"> : Delete Disk from Pool - list : Display disk Pool Info - [options] - -P : Disk Pool Id - -D : Disk Id - -N : Disk Pool Name - -n : Disk Name - -r : Replica Count. 1~3 - -t : Disk Pool Class Type. 'STANDARD' or 'ARCHIVE' - -c : Disk Pool Description - -l : Disk Pool Detail information - -h, --help : show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {init|add|remove|update|show|status} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-P', "--Id", dest="DiskPoolId", help='Disk Pool Id') - parser.add_option('-N', "--DiskPoolName", dest="DiskPoolName", help='Disk Pool Name') - parser.add_option('-n', "--DiskName", dest="DiskName", help='Disk Name') - parser.add_option('-D', "--DiskIds", dest="DiskIds", help='Disk Id List to Add to Pool') - parser.add_option('-c', "--Description", dest="Description", help='Description of Pool') - parser.add_option('-r', "--Nreplica", dest="replica", type='int', help='Replica Count. 1~3') - parser.add_option('-t', "--ClassType", dest="DiskPoolType", type='string', help='Disk Pool Type. \'STANDARD\' or \'ARCHIVE\'') - parser.add_option('-l', "--Detail", dest="Detail", action='store_true', help='Detail info') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='Debug mode') - - options, args = parser.parse_args() - - if len(args) < 1: - parser.print_help() - sys.exit(-1) - - IfsPortalIp = '127.0.0.1' - IfsPortalPort = 5443 - IfsMqPort = 5672 - IfsPortalKey = '' - ret, conf = read_conf(MonServicedConfPath) - if ret is True: - IfsPortalIp = conf['mgs']['MgsIp'] - IfsPortalPort = int(conf['mgs']['IfsPortalPort']) - IfsMqPort = int(conf['mgs']['MqPort']) - IfsPortalKey = conf['mgs']['IfsPortalKey'] - - options.IpAddress = IfsPortalIp - options.Port = IfsPortalPort - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - - """ - ServerId = None - if args[0] in ['add', 'delete']: - Res, Errmsg, Ret, AllDiskList = GetDiskInfo(IfsPortalIp, IfsPortalPort, DiskId=options.DiskId, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - if len(AllDiskList) == 0: - print('ServerId is not found with DiskId %s' % options.DiskId) - sys.exit(-1) - else: - Res, Errmsg, Svr = GetServerInfoWithDiskId(AllDiskList, options.DiskId) - if Res == ResOk: - ServerId = Svr.Id - else: - print('fail to get Server Id with DiskId: %s' % Errmsg) - sys.exit(-1) - - else: - print('fail to get ServerId with DiskID') - sys.exit(-1) - """ - if args[0] == 'add': - if not (options.DiskPoolName and options.replica and options.DiskPoolType): - parser.print_help() - sys.exit(-1) - if options.replica not in [1, 2, 3] or options.DiskPoolType not in [DiskPoolClassStandard, DiskPoolClassArchive]: - parser.print_help() - sys.exit(-1) - if options.replica == 2: - ReplicationType = DiskPoolReplica2 - elif options.replica == 3: - ReplicationType = DiskPoolReplica3 - else: - ReplicationType = DiskPoolReplica1 - - Res, Errmsg, Ret = AddDiskPool(IfsPortalIp, IfsPortalPort, IfsPortalKey, options.DiskPoolName, options.DiskPoolType, - ReplicationType, Description=options.Description, - logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'remove': - if not (options.DiskPoolId or options.DiskPoolName): - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = RemoveDiskPool(IfsPortalIp, IfsPortalPort, IfsPortalKey, DiskPoolId=options.DiskPoolId, DiskPoolName=options.DiskPoolName, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] in ['addDisk', 'removeDisk', 'update']: - AddDiskIds = None - DelDiskIds = None - if args[0] == 'addDisk': - if not ((options.DiskPoolId and options.DiskIds) or (options.DiskPoolName and options.DiskName)): - parser.print_help() - sys.exit(-1) - if options.DiskIds is not None: - AddDiskIds = options.DiskIds.split() - elif options.DiskName is not None: - AddDiskIds = options.DiskName.split() - else: - parser.print_help() - print("Disk Ids or Disk Name is required") - sys.exit(-1) - elif args[0] == 'removeDisk': - if not ((options.DiskPoolId and options.DiskIds) or (options.DiskPoolName and options.DiskName)): - parser.print_help() - sys.exit(-1) - DelDiskIds = options.DiskIds.split() - - Res, Errmsg, Ret = UpdateDiskPool(IfsPortalIp, IfsPortalPort, IfsPortalKey, DiskPoolId=options.DiskPoolId, AddDiskIds=AddDiskIds, DelDiskIds=DelDiskIds, - DiskPoolName=options.DiskPoolName, Description=options.Description, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'list': - Res, Errmsg, Ret, DiskPoolList = GetDiskPoolInfo(options.IpAddress, options.Port, IfsPortalKey, DiskPoolId=options.DiskPoolId, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - Res, Errmsg, Ret, ServerDetailInfo = GetAllServerDetailInfo(IfsPortalIp, IfsPortalPort, IfsPortalKey, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - if options.Detail: - ShowDiskPoolInfo(DiskPoolList, ServerDetailInfo, Detail=True) - else: - ShowDiskPoolInfo(DiskPoolList, ServerDetailInfo) - else: - parser.print_help() diff --git a/core/mgs/util/ksanManager b/core/mgs/util/ksanManager deleted file mode 100755 index 98791c9d..00000000 --- a/core/mgs/util/ksanManager +++ /dev/null @@ -1,147 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os, sys -import pdb - -if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: - sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import read_conf -from user.user_manage import * -#from ksan.volume.volume_manage import * -from disk.disk_manage import * -from optparse import OptionParser - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ifs_user {add|remove|start|stop|set|list} [option] - add -n -i - -e -u - : Add user - remove -U : Remove user - update -U : Update User Info - list -U : Display User Info - [options] - -h, --help : Show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {init|add|remove|update|show|status} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-i', "--LoginId", dest="LoginId", help='User Login Id') - parser.add_option('-U', "--UserId", dest="UserId", help='User Id') - parser.add_option('-n', "--Name", dest="Name", help='User Name') - parser.add_option('-e', "--Email", dest="Email", help='User Email Info') - parser.add_option('-w', "--Password", dest="Password", help='User Password') - parser.add_option('-l', "--Detail", dest="Detail", action='store_true', help='Detail info') - parser.add_option('-u', "--Roles", dest="Roles", help='User Roles') - #parser.add_option('-s', "--Status", dest="Status", help='User Status') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='Debug mode') - - options, args = parser.parse_args() - - if len(args) < 1: - parser.print_help() - sys.exit(-1) - - IfsPortalIp = '127.0.0.1' - IfsPortalPort = 5443 - IfsMqPort = 5672 - ret, conf = read_conf(MonServicedConfPath) - if ret is True: - IfsPortalIp = conf['mgs']['MgsIp'] - IfsPortalPort = int(conf['mgs']['IfsPortalPort']) - IfsMqPort = int(conf['mgs']['MqPort']) - - options.IpAddress = IfsPortalIp - options.Port = IfsPortalPort - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - - if args[0] == 'add': - if not options.Roles: - print('User Roles Info is required') - sys.exit(-1) - - options.Roles = options.Roles.split() - Res, Errmsg, Ret = AddUser(IfsPortalIp, IfsPortalPort, options.LoginId, options.Name, options.Roles, Email=options.Email, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'remove': - if not options.UserId: - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = RemoveUser(IfsPortalIp, IfsPortalPort, options.UserId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'update': - if not options.UserId: - parser.print_help() - sys.exit(-1) - if options.Roles: - options.Roles = options.Roles.split() - Res, Errmsg, Ret = UpdateUserInfo(IfsPortalIp, IfsPortalPort, options.UserId, Name=options.Name, Email=options.Email, - Roles=options.Roles, Status=options.Status, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'addRoles': - if not options.UserId: - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = AddUserRoles(IfsPortalIp, IfsPortalPort, options.UserId, options.Roles, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'removeRoles': - if not (options.UserId and options.Roles): - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = RemoveUserRoles(IfsPortalIp, IfsPortalPort, options.UserId, options.Roles, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'changePassword': - if not (options.UserId and options.Password): - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = ChangeUserPassword(IfsPortalIp, IfsPortalPort, options.UserId, options.Password, - options.Password, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - elif args[0] == 'list': - Res, Errmsg, Ret, Users = GetUserInfo(options.IpAddress, options.Port, UserId=options.UserId, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - if Ret.Result != ResultSuccess: - print(Ret.Message) - else: - ShowUserInfo(Users, Detail=options.Detail) - else: - parser.print_help() diff --git a/core/mgs/util/ksanNetwork b/core/mgs/util/ksanNetwork deleted file mode 100755 index ee847842..00000000 --- a/core/mgs/util/ksanNetwork +++ /dev/null @@ -1,194 +0,0 @@ -#!/bin/env python3 -""" -* Copyright 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE.md for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 -""" - -import os -import sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import read_conf -from network.network_manage import * -from server.server_manage import * - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanNetwork {add|remove|update|show|status|exists_check} [option] - init : init management server ip and port - add -n -S : register nic to the management network interface pool - addvlan -V -i -m -g : register vlan network to the vlan network interface pool - remove -S : unregister local server from management servers pool - remove_vlan -S -N -V : unregister vlan network from vlan network pool - update -n -N -S : update specific nic info - update_status -N -S : update specific network interface status - list -S : show the registered network interface info with server id - check -S -n -N : check if a regisetered network interface exists with nic name. exclude with nic id - check_vlan -S -n -N -V : check if a regisetered vlan network interface exists with nic name. exclude with nic id - [options] - -S : Server Id - -n : Network Device Name - -N : Network Device Id - -V : Vlan Id - -i : Ip Address - -m : Nework SubnetMask - -g : Gateway Ip - -h, --help : show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {init|add|remove|update|show|status} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-i', "--Ip", dest="IpAddress", help='IpAddress') - parser.add_option('-m', "--SubnetMask", dest="SubnetMask", help='SubnetMask') - parser.add_option('-g', "--Gateway", dest="Gateway", help='Gateway') - parser.add_option('-V', "--VlanInterfceid", dest="Tag", help='Vlan Network id') - parser.add_option('-I', "--InterfaceId", dest="InterfaceId", help='Network Interface Id') - parser.add_option('-S', "--Serverid", dest="ServerId", help='Server Id') - parser.add_option('-v', "--Vlanid", dest="VlanId", help='Vlan Id') - parser.add_option('-c', "--Description", dest="Description", default='', help='server description') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='debug mode') - parser.add_option('-n', "--Name", dest="Name", help='Network Interface Name') - parser.add_option('-l', "--Detail", dest="Detail", action='store_true', help='Network Info in Detail') - - options, args = parser.parse_args() - - if len(args) != 1: - parser.print_help() - sys.exit(-1) - - IfsPortalIp = '127.0.0.1' - IfsPortalPort = 5443 - IfsMqPort = 5672 - ret, conf = read_conf(MonServicedConfPath) - if ret is True: - IfsPortalIp = conf['mgs']['MgsIp'] - IfsPortalPort = int(conf['mgs']['IfsPortalPort']) - IfsMqPort = int(conf['mgs']['MqPort']) - - - - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - if args[0] == 'add': - if not (options.ServerId and options.Name): - parser.print_help() - else: - Res, Errmsg, Ret = AddNetworkInterface(IfsPortalIp, IfsPortalPort, options.ServerId, options.Name, Description=options.desc, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'addvlan': - if not (options.ServerId and options.NicId and options.Tag and options.IpAddress and options.SubnetMask and options.Gateway): - parser.print_help() - else: - Tag = options.Tag - IpAddress = options.IpAddress - SubnetMask = options.SubnetMask - Gateway = options.Gateway - Res, Errmsg, Ret = add_vlan_network_interface(IfsPortalIp, IfsPortalPort, options.ServerId, options.NicId, Tag, IpAddress, SubnetMask, Gateway, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'update': - if not options.InterfaceId: - parser.print_help() - else: - Res, Errmsg, Ret = UpdateNetworkInterfaceInfo(IfsPortalIp, IfsPortalPort, options.InterfaceId, - Name=options.Name, Description=options.Description, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'update_vlan': - if not (options.ServerId and options.NicId and options.VlanId and options.Tag and options.IpAddress and options.SubnetMask and options.Gateway): - parser.print_help() - else: - Res, Errmsg, Ret = update_vlan_network_interface_info(IfsPortalIp, IfsPortalPort, options.ServerId, options.NicId, options.VlanId, options.Tag, options.IpAddress, options.SubnetMask, options.Gateway, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - elif args[0] == 'update_status': - if not (options.ServerId and options.NicId): - parser.print_help() - else: - Res, Errmsg, Ret = UpdateNetworkInterfaceLinkState(IfsPortalIp, IfsPortalPort, options.ServerId, options.NicId, "Down", logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'remove': - if not (options.InterfaceId): - parser.print_help() - else: - Res, Errmsg, Ret = DeleteNetworkInterface(IfsPortalIp, IfsPortalPort, options.InterfaceId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'remove_vlan': - if not (options.ServerId and options.NicId): - parser.print_help() - else: - Res, Errmsg, Ret = delete_vlan_network_interface(IfsPortalIp, IfsPortalPort, options.ServerId, options.NicId, options.VlanId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'list': - #Res, Errmsg, Ret, Data = GetNetworkInterface(IfsPortalIp, IfsPortalPort, ServerId=options.ServerId, NicId=options.NicId, disp=True, - # logger=logger) - Res, Errmsg, Ret, Data = GetAllServerDetailInfo(IfsPortalIp, IfsPortalPort, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - ShowNetworkInterfaceInfo(Data, InterfaceId=options.InterfaceId, Detail=options.Detail) - else: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'show_vlan': - if not (options.ServerId and options.NicId): - parser.print_help() - else: - Res, Errmsg, Ret = GetVlanNetworkInterfaces(IfsPortalIp, IfsPortalPort, options.ServerId, options.NicId, VlanId=options.VlanId, disp=True, - logger=logger) - if Res != ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'check': - if not (options.ServerId and options.Name): - parser.print_help() - else: - Res, Errmsg, Ret = check_exist_network_interface(IfsPortalIp, IfsPortalPort, options.ServerId, options.Name, options.NicId, logger=logger) - if Res == ResOk: - print(Ret) - else: - print(Errmsg) - elif args[0] == 'check_vlan': - if not (options.ServerId and options.NicId and options.Tag): - parser.print_help() - else: - Res, Errmsg, Ret = check_exist_vlan_network_interface(IfsPortalIp, IfsPortalPort, options.ServerId, options.NicId, options.Tag, VlanId=options.VlanId, logger=logger) - if Res == ResOk: - print(Ret) - else: - print(Errmsg) - else: - parser.print_help() - diff --git a/core/mgs/util/ksanServer b/core/mgs/util/ksanServer deleted file mode 100755 index c8c6e1c1..00000000 --- a/core/mgs/util/ksanServer +++ /dev/null @@ -1,136 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os, sys -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import read_conf -from server.server_manage import * -import pdb - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanServer {add|remove|update|list} [option] - add -i : Add Server to servers pool - remove -N : Remove server from servers pool - update -N -c : Update server info - list -l : Show the registered server info - [options] - -S : Server Id - -N : Server Name - -e : Description - -l : Show info in detail - -h, --help : show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {add|remove|update|list} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-S', "--Id", dest="ServerId", help='Server Id') - parser.add_option('-i', "--IpAddress", dest="IpAddress", default='', help='Server Ip Address') - parser.add_option('-c', "--Description", dest="Description", default='', help='Server Description') - parser.add_option('-N', "--Name", dest="Name", default=None, help='Server Name') - parser.add_option('-l', "--Detail", dest="Detail", action='store_true', help='Server Info in detail') - parser.add_option('-s', "--State", dest="State", help='Server State') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='debug mode') - #parser.add_option('-A', "--LoadAverage", dest="LoadAverage", default=False, help='LoadAverage') - - options, args = parser.parse_args() - if len(args) != 1: - parser.print_help() - sys.exit(-1) - - IfsPortalIp = '' - IfsPortalPort = 0 - IfsMqPort = 0 - ApiKey = '' - ret, conf = read_conf(MonServicedConfPath) - if ret is True: - IfsPortalIp = conf['mgs']['MgsIp'] - IfsPortalPort = int(conf['mgs']['IfsPortalPort']) - IfsMqPort = int(conf['mgs']['MqPort']) - ApiKey = conf['mgs']['IfsPortalKey'] - else: - if args[0] != 'init': - print("ksanMonitor.conf is not configured. ./ksanServer init first") - sys.exit(-1) - - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - - if args[0] == 'add': - Res, Errmsg, Ret = ServerInit(IfsPortalIp, IfsPortalPort, options.IpAddress, IfsMqPort, ApiKey, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'remove': - if not (options.ServerId or options.Name): - parser.print_help() - sys.exit(-1) - - Res, Errmsg, Ret = RemoveServer(IfsPortalIp, IfsPortalPort, ApiKey, ServerId=options.ServerId, Name=options.Name, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'update': - if not (options.ServerId or options.Name): - parser.print_help() - sys.exit(-1) - - Res, Errmsg, Ret = UpdateServerInfo(IfsPortalIp, IfsPortalPort, ApiKey, ServerId=options.ServerId, Name=options.Name, - Description=options.Description, State=options.State, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - elif args[0] == 'update_state': - if options.ServerId is None: - parser.print_help() - sys.exit(-1) - - Res, Errmsg, Ret = UpdateServerInfo(IfsPortalIp, IfsPortalPort, ApiKey, ServerId=options.ServerId, Name=options.Name, State=options.State, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'list': - Detail = False - if options.Detail: - Detail = True - Res, errmsg, Ret, Data = GetAllServerDetailInfo(IfsPortalIp, IfsPortalPort, ApiKey, logger=logger) - if Res == ResOk: - if Ret.Result == ResultSuccess: - ShowServerInfo(Data, Id=options.ServerId, Detail=options.Detail) - else: - print(Ret.Result, Ret.Message) - else: - print(errmsg) - elif args[0] == 'check': - if options.name is None: - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = server_exist_check_with_servername(IfsPortalIp, IfsPortalPort, options.Name, ApiKey, - ExcludeServerId=options.ServerId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message, Ret.Data) - else: - print(Errmsg) - else: - parser.print_help() diff --git a/core/mgs/util/ksanService b/core/mgs/util/ksanService deleted file mode 100755 index 3d462d7c..00000000 --- a/core/mgs/util/ksanService +++ /dev/null @@ -1,243 +0,0 @@ -#!/usr/bin/env python3 -# -*- encoding: utf-8 -*- -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import sys, os -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from service.service_manage import * -from server.server_manage import * -from common.init import GetConf -from common.define import TypeServiceMongoDB, TypeServiceS3, TypeServiceOSD, TypeServiceHaproxy, \ - TypeServiceMonitor, TypeServiceAgent - - -class MyOptionParser(OptionParser): - def print_help(self): - """ add -S -n -G - -T : Add Service - """ - - Usage = """ - Usage: ksanService {add|remove|config|start|stop|update|list} [option] - add -n -N -t : Add Service - remove -N : Delete Service - config {list|get|add|update_version|remove} -t - -v : Config Service Conf - start -N : Start Service - stop -N : Stop Service - list : Display Service info - [options] - -S : Server Id - -I : Service Id - -N : Service Name - -G : Service Group Id - -t : Service Type. - -l : Show Service Info in detail - -h, --help : show this help message and exit -""" - print(Usage) - - -def ConfigControl(PortalIp, PortalPort, ApiKey, args, options, logger): - if len(args) != 2: - parser.print_help() - sys.exit(-1) - if not options.ServiceType: - print('Service type(-t) is required') - sys.exit(-1) - if options.ServiceType not in [TypeServiceMongoDB, TypeServiceOSD, TypeServiceMonitor, TypeServiceGW, - TypeServiceMonitor, TypeServiceAgent]: - print('Invalid Service Type. Valid Service Type: %s' % ', '.join([TypeServiceMongoDB, TypeServiceOSD, - TypeServiceMonitor, TypeServiceGW, TypeServiceMonitor, TypeServiceAgent])) - sys.exit(-1) - if args[1] == 'add': - ServiceType = options.ServiceType - if options.File: - ConfigFile = options.File - else: - ConfigFile = None - Res, Errmsg, Ret = SetServiceConfig(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, ConfigFilePath=ConfigFile, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - print(Ret.Message) - - elif args[1] == 'get': - Version = options.version - File = options.File - ServiceType = options.ServiceType - Res, Errmsg, Ret, Data = GetServiceConfig(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, Version=Version, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - if(Ret.Result == ResultSuccess): - DsPServiceConf(Data) - else: - print(Ret.Message) - - elif args[1] == 'update_version': - if not options.version: - parser.print_help() - print('config version is required.') - sys.exit(-1) - Version = options.version - ServiceType = options.ServiceType - Res, Errmsg, Ret = UpdateServiceConfigVersion(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, Version, logger=logger) - print(Ret.Result, Ret.Message) - - elif args[1] == 'remove': - Version = options.version - ServiceType = options.ServiceType - Res, Errmsg, Ret = RemoveServiceConfig(IfsPortalIp, IfsPortalPort, ApiKey, ServiceType, Version, logger=None) - print(Ret.Result, Ret.Message) - - elif args[1] == 'list': - Version = options.version - ServiceType = options.ServiceType - Res, Errmsg, Ret, Data = GetServiceConfigList(PortalIp, PortalPort, ApiKey, ServiceType, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - print(Ret.Result, Ret.Message) - ShowConfigList(Data) - else: - parser.print_help() - sys.exit(-1) - - -if __name__ == '__main__': - usage = "Usage: %prog {add|remove|config|start|stop|update|list} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-S', "--ServerId", dest="ServerId", help='Server Id') - parser.add_option('-I', "--ServiceId", dest="ServiceId", help='Service Id') - parser.add_option('-N', "--ServiceName", dest="ServiceName", help='Service Name') - parser.add_option('-n', "--ServerName", dest="ServerName", help='Server Name') - parser.add_option('-G', "--GroupId", dest="GroupId", help='Group Id') - parser.add_option('-t', "--ServiceType", dest="ServiceType", help='Service Type') - parser.add_option('-F', "--File", dest="File", help='Config File Path') - #parser.add_option('-V', "--VlanIds", dest="VlanIds", help='Vlan Id') - parser.add_option('-v', "--version", dest="version", help='version') - parser.add_option('-c', "--Description", dest="Description", help='Description') - #parser.add_option('-s', "--State", dest="State", help='State') - parser.add_option('-l', "--Detail", dest="Detail", action='store_true', help='server info in detail') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='debug mode') - - options, args = parser.parse_args() - - IfsPortalIp = '127.0.0.1' - IfsPortalPort = 5443 - IfsMqPort = 5672 - IfsPortalKey = '' - ret, conf = GetConf(MonServicedConfPath) - if ret is True: - IfsPortalIp = conf.mgs.MgsIp - IfsPortalPort = int(conf.mgs.IfsPortalPort) - IfsMqPort = int(conf.mgs.MqPort) - IfsPortalKey = conf.mgs.IfsPortalKey - - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - else: - logger = Logging(loglevel='error') - logger = logger.create() - - if len(args) < 1: - parser.print_help() - sys.exit(-1) - - if args[0] == 'add': - if not ((options.ServerName or options.ServerId) and options.ServiceName and options.ServiceType): - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = AddService(IfsPortalIp, IfsPortalPort, IfsPortalKey, options.ServiceName, options.ServiceType, ServerId=options.ServerId, - ServerName=options.ServerName, GroupId=options.GroupId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'remove': - if not (options.ServiceId or options.ServiceName): - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = DeleteService(IfsPortalIp, IfsPortalPort, IfsPortalKey, ServiceId=options.ServiceId, - ServiceName=options.ServiceName, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'update': - if not (options.ServiceId or options.ServiceName): - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = UpdateServiceInfo(IfsPortalIp, IfsPortalPort, IfsPortalKey, ServiceId=options.ServiceId, ServiceName=options.ServiceName, - ServiceType=options.ServiceType, - GroupId=options.GroupId, Description=options.Description, - logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - elif args[0] == 'update_state': - Res, Errmsg, Ret = UpdateServiceState(IfsPortalIp, IfsPortalPort, IfsPortalKey, options.ServiceId, - logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - elif args[0] == 'update_usage': - Res, Errmsg, Ret = UpdateServiceUsage(IfsPortalIp, IfsPortalPort, IfsPortalKey, options.ServiceId, - logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'config': - ConfigControl(IfsPortalIp, IfsPortalPort, IfsPortalKey, args, options, logger) - - """ - Res, Errmsg, Ret = UpdateServiceConf(IfsPortalIp, IfsPortalPort, options.ServiceId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - """ - elif args[0] in ['start', 'stop']: - if not (options.ServiceId or options.ServiceName): - parser.print_help() - sys.exit(-1) - - Res, Errmsg, Ret = ControlService(IfsPortalIp, IfsPortalPort, IfsPortalKey, args[0], ServiceId=options.ServiceId, - ServiceName=options.ServiceName, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'list': - if options.Detail: - Res, Errmsg, Ret, Data = GetAllServerDetailInfo(IfsPortalIp, IfsPortalPort, IfsPortalKey,logger=logger) - else: - Res, Errmsg, Ret, Data = GetServiceInfo(IfsPortalIp, IfsPortalPort, IfsPortalKey, logger=logger) - - if Res == ResOk: - print(Ret.Result, Ret.Message) - if options.Detail: - ShowServiceInfoWithServerInfo(Data, Detail=options.Detail) - #ShowServiceInfo(Data, Detail=options.Detail) - else: - ShowServiceInfo(Data, Detail=options.Detail) - else: - print(Errmsg) - else: - parser.print_help() diff --git a/core/mgs/util/ksanServicegroup b/core/mgs/util/ksanServicegroup deleted file mode 100755 index c2830e2f..00000000 --- a/core/mgs/util/ksanServicegroup +++ /dev/null @@ -1,146 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import sys, os -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from service.service_manage import * -from server.server_manage import * -from common.init import GetConf - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ifs_serviceGroup {add|remove|update|start|stop|addService|removeService|list} [option] - add -n -t : Add Group - remove -G : Delete Group - update -G : Update Group Info - start -G : Start Group - stop -G : Stop Group - addService -G -I : Add Service to Group - removeService -G -I : Remove Service to Group - list -l : Show Group info - [options] - -I : Service Id - -n : Service Name - -G : Service Group Id - -t : Service Type - -l : Show Service Info in detail - -h, --help : show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {add|remove|update|start|stop|addService|removeService|list} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-S', "--ServerId", dest="ServerId", help='Server Id') - parser.add_option('-I', "--ServiceId", dest="ServiceId", help='Service Id') - parser.add_option('-n', "--Name", dest="Name", help='server description') - parser.add_option('-G', "--GroupId", dest="GroupId", help='Group Id') - parser.add_option('-D', "--ServiceIds", dest="ServiceIds", default=[],help='Service Ids') - parser.add_option('-t', "--ServiceType", dest="ServiceType", help='Service Type') - parser.add_option('-c', "--Description", dest="Description", help='Description') - parser.add_option('-l', "--Detail", dest="Detail", action='store_true', help='server info in detail') - parser.add_option('-i', "--Ipaddr", dest="IpAddress", help='Service Group IpAddress') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='debug mode') - - options, args = parser.parse_args() - if len(args) != 1: - parser.print_help() - sys.exit(-1) - - IfsPortalIp = '127.0.0.1' - IfsPortalPort = 5443 - IfsMqPort = 5672 - ret, conf = GetConf(MonServicedConfPath) - if ret is True: - IfsPortalIp = conf.mgs.MgsIp - IfsPortalPort = int(conf.mgs.IfsPortalPort) - IfsMqPort = int(conf.mgs.MqPort) - - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - - if args[0] == 'add': - if options.ServiceType not in ['OSD', 'GW']: - parser.print_help() - sys.exit(-1) - - Res, Errmsg, Ret = AddServiceGroup(IfsPortalIp, IfsPortalPort, options.Name, options.ServiceType, - Description=options.Description, ServiceIpAddress=options.IpAddress, - ServiceIds=options.ServiceIds, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] == 'remove': - Res, Errmsg, Ret = RemoveServiceGroup(IfsPortalIp, IfsPortalPort, options.GroupId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] in ['addService', 'removeService', 'update']: - AddServiceIds = None - removeServiceIds = None - if args[0] == 'addService': - if not (options.ServiceId and options.GroupId): - parser.print_help() - sys.exit(-1) - AddServiceIds = options.ServiceId.split() - elif args[0] == 'removeService': - if not (options.ServiceId and options.GroupId): - parser.print_help() - sys.exit(-1) - removeServiceIds = options.ServiceId.split() - Res, Errmsg, Ret = UpdateServiceGroup(IfsPortalIp, IfsPortalPort, options.GroupId, Name=options.Name, - Description=options.Description, - addServiceIds=AddServiceIds,removeServiceIds=removeServiceIds, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - elif args[0] == 'config': - Res, Errmsg, Ret = UpdateServiceConf(IfsPortalIp, IfsPortalPort, options.ServiceId, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - elif args[0] in ['start', 'stop', 'restart']: - if not options.GroupId: - parser.print_help() - sys.exit(-1) - Action = 'Start' - if args[0] == 'stop': - Action = 'Stop' - elif args[0] == 'restart': - Action = 'Restart' - - Res, Errmsg, Ret = ControlSerivceGroup(IfsPortalIp, IfsPortalPort, options.GroupId, Action, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - else: - print(Errmsg) - - - elif args[0] == 'list': - Res, Errmsg, Ret, Data = GetAllServiceGroups(IfsPortalIp, IfsPortalPort, logger=logger) - if Res == ResOk: - print(Ret.Result, Ret.Message) - ShowServiceGroup(Data, Detail=options.Detail) - else: - print(Errmsg) - else: - parser.print_help() diff --git a/core/mgs/util/ksanSysinfo b/core/mgs/util/ksanSysinfo deleted file mode 100755 index 51bb0e6e..00000000 --- a/core/mgs/util/ksanSysinfo +++ /dev/null @@ -1,78 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - - -import os, sys -import pdb -sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from disk.disk_manage import * -from network.network_manage import * -from common.init import * - -def main(): - # Server info - ret, conf = GetConf(MonServicedConfPath) - if ret is False: - print('Check if configuration is done') - sys.exit(-1) - - MgsIp = conf.mgs.MgsIp - PortalPort = int(conf.mgs.IfsPortalPort) - Res, Errmsg, Ret, AllServers = GetServerInfo(MgsIp, PortalPort) - if Res != ResOk: - print(Errmsg) - sys.exit(-1) - - Res, Errmsg, Ret, AllDisks = GetDiskInfo(MgsIp, int(PortalPort), conf.mgs.IfsPortalKey) - if Res != ResOk: - print(Errmsg) - - ServerNetworks = dict() - for Svr in AllServers: - Res, Errmsg, Ret, Networks = GetNetworkInterface(MgsIp, PortalPort, ServerId=Svr.Id) - if Res != ResOk: - print(Errmsg) - else: - ServerNetworks[Svr.Id] = list() - for Network in Networks: - ServerNetworks[Svr.Id].append(Network) - - ShowInfo(AllServers, AllDisks, ServerNetworks) - - - -def main2(): - # Server info - ret, conf = GetConf(MonServicedConfPath) - if ret is False: - print('Check if configuration is done') - sys.exit(-1) - - AllServersDetail = list() - MgsIp = conf.mgs.MgsIp - PortalPort = int(conf.mgs.IfsPortalPort) - Res, Errmsg, Ret, AllServers = GetServerInfo(MgsIp, PortalPort) - if Res != ResOk: - print(Errmsg) - return AllServersDetail - - for Svr in AllServers: - Res, Errmsg, Ret, ServerDetail = GetServerInfo(MgsIp, PortalPort, ServerId=Svr.Id) - if Res != ResOk: - print(Errmsg) - else: - AllServersDetail.append(ServerDetail) - return AllServersDetail - - -AllserversDetail = main2() -ShowSysInfo(AllserversDetail) diff --git a/core/mgs/util/ksanUser b/core/mgs/util/ksanUser deleted file mode 100755 index fc6720ad..00000000 --- a/core/mgs/util/ksanUser +++ /dev/null @@ -1,153 +0,0 @@ -#!/usr/bin/env python3 -""" -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" - -import os, sys -import pdb - -if os.path.dirname(os.path.abspath(os.path.dirname(__file__))) not in sys.path: - sys.path.append(os.path.dirname(os.path.abspath(os.path.dirname(__file__)))) -from common.init import read_conf, get_input -from user.user_manage import * -from disk.disk_manage import * -from optparse import OptionParser - - -class MyOptionParser(OptionParser): - def print_help(self): - Usage = """ - Usage: ksanUser {add|remove|update|list} [option] - add -N -n -e : Add S3user - remove -N : Remove S3user - addStorageclass -N -n -s : Add S3user Storage Class - update -N -e : Update S3User Info - list : Display S3User Info - [options] - -U : S3User Id - -N : S3User Name - -n : Disk Pool Name - -e : Email Address - -w : S3User password - -l : S3User Detail information - -h, --help : Show this help message and exit -""" - print(Usage) - - -if __name__ == '__main__': - usage = "Usage: %prog {add|remove|update|list} [option]" - parser = MyOptionParser(usage=usage) - parser.add_option('-U', "--UserId", dest="UserId", help='S3 User Id') - parser.add_option('-P', "--DiskPoolId", dest="DiskPoolId", help='Disk Pool Id') - parser.add_option('-N', "--UserName", dest="UserName", help='User Name') - parser.add_option('-n', "--DiskPoolName", dest="DiskPoolName", help='Disk Pool Name') - parser.add_option('-s', "--StorageClass", dest="StorageClass", help='Storage Class') - parser.add_option('-e', "--Email", dest="Email", help='User Email Info') - parser.add_option('-w', "--Password", dest="Password", help='User Password') - parser.add_option('-l', "--Detail", dest="Detail", action='store_true',default=False, help='Detail info') - parser.add_option('-d', "--Debug", dest="debug", action='store_true', default=False, help='Debug mode') - - options, args = parser.parse_args() - - if len(args) < 1: - parser.print_help() - sys.exit(-1) - - IfsPortalIp = '127.0.0.1' - IfsPortalPort = 5443 - IfsMqPort = 5672 - IfsPortalKey = '' - ret, conf = read_conf(MonServicedConfPath) - if ret is True: - IfsPortalIp = conf['mgs']['MgsIp'] - IfsPortalPort = int(conf['mgs']['IfsPortalPort']) - IfsMqPort = int(conf['mgs']['MqPort']) - IfsPortalKey = conf['mgs']['IfsPortalKey'] - - options.IpAddress = IfsPortalIp - options.Port = IfsPortalPort - logger = None - if options.debug is True: - logger = Logging(loglevel='debug') - logger = logger.create() - - if args[0] == 'add': - if not (options.UserName and options.Email and (options.DiskPoolId or options.DiskPoolName)): - print('User Name, Default DiskPoolName and Email Info are required') - sys.exit(-1) - - Res, Errmsg, Ret = AddS3User(IfsPortalIp, IfsPortalPort, IfsPortalKey, options.UserName, DiskPoolId=options.DiskPoolId, - DiskPoolName=options.DiskPoolName, Email=options.Email, logger=logger) - if Res == ResOk: - print(Ret.Result) - else: - print(Errmsg) - elif args[0] == 'remove': - if not (options.UserId or options.UserName): - parser.print_help() - sys.exit(-1) - - Conf = dict() - Conf['isRemove']= get_input('Are you sure to remove user(yes|no)?', str, 'no', ValidAnsList=['yes', 'no']) - if Conf['isRemove'] == 'yes': - Res, Errmsg, Ret = RemoveS3User(IfsPortalIp, IfsPortalPort, IfsPortalKey, UserId=options.UserId, UserName=options.UserName, logger=logger) - if Res == ResOk: - print(Ret.Result) - else: - print(Errmsg) - - elif args[0] == 'addStorageclass': - if not ((options.UserId or options.UserName) and (options.DiskPoolId or options.DiskPoolName) and options.StorageClass): - print('User Name, Default DiskPoolName and Email Info are required') - sys.exit(-1) - - Res, Errmsg, Ret = AddS3UserStorageClass(IfsPortalIp, IfsPortalPort, IfsPortalKey ,options.UserName, DiskPoolId=options.DiskPoolId, - DiskPoolName=options.DiskPoolName, Email=options.Email, logger=logger) - if Res == ResOk: - print(Ret.Result) - else: - print(Errmsg) - - elif args[0] == 'removeStorageclass': - if not ((options.UserId or options.UserName) and (options.DiskPoolId or options.DiskPoolName) and options.StorageClass): - print('User Name, Default DiskPoolName and Email Info are required') - sys.exit(-1) - - Res, Errmsg, Ret = RemoveS3UserStorageClass(IfsPortalIp, IfsPortalPort, IfsPortalKey, options.UserName, DiskPoolId=options.DiskPoolId, - DiskPoolName=options.DiskPoolName, Email=options.Email, logger=logger) - if Res == ResOk: - print(Ret.Result) - else: - print(Errmsg) - - - elif args[0] == 'update': - if not (options.UserId or options.UserName): - parser.print_help() - sys.exit(-1) - Res, Errmsg, Ret = UpdateS3UserInfo(IfsPortalIp, IfsPortalPort, IfsPortalKey, UserId=options.UserId, UserName=options.UserName, - Email=options.Email, logger=logger) - if Res == ResOk: - print(Ret.Result) - else: - print(Errmsg) - - elif args[0] == 'list': - Res, Errmsg, Ret, Users = GetS3UserInfo(options.IpAddress, options.Port, IfsPortalKey, UserId=options.UserId, logger=logger) - if Res != ResOk: - print(Errmsg) - else: - if Ret.Result != ResultSuccess: - print(Ret.Message) - else: - ShowS3UserInfo(Users, Detail=options.Detail) - else: - parser.print_help() diff --git a/core/mgs/util/setup.py b/core/mgs/util/setup.py index 8638c98e..f8e19531 100755 --- a/core/mgs/util/setup.py +++ b/core/mgs/util/setup.py @@ -12,6 +12,10 @@ Version = '0.8.0' +os.system("rm -f /usr/local/ksan/bin/ksan") +os.system("rm -f /usr/local/ksan/bin/util/ksanCbalance.jar") +os.system("rm -f /usr/local/ksan/bin/util/ksanFsck.jar") +os.system("rm -f /usr/local/ksan/bin/util/ksanGetAttr.jar") setup( name='ksan-mgs-util', diff --git a/core/src/com/pspace/ifs/ksan/gw/GWMain.java b/core/src/com/pspace/ifs/ksan/gw/GWMain.java index ba86fa66..d372ccf6 100644 --- a/core/src/com/pspace/ifs/ksan/gw/GWMain.java +++ b/core/src/com/pspace/ifs/ksan/gw/GWMain.java @@ -14,10 +14,13 @@ import java.io.FileWriter; import java.io.IOException; import java.io.PrintStream; +import java.util.TimeZone; +import java.util.Locale; import java.io.BufferedReader; import java.io.FileReader; import com.pspace.ifs.ksan.gw.handler.GW; +import com.pspace.ifs.ksan.gw.handler.Azu; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.libs.HeartbeatManager; import com.pspace.ifs.ksan.libs.PrintStack; @@ -34,6 +37,7 @@ public class GWMain { private static GW gw; + private static final Logger logger = LoggerFactory.getLogger(GWMain.class); private GWMain() { @@ -51,6 +55,32 @@ public static void main(String[] args) throws Exception { Runtime.getRuntime().addShutdownHook(new HookThread()); writePID(); + // setting timezone, locale + TimeZone.setDefault(TimeZone.getTimeZone("Asia/Seoul")); + Locale.setDefault(Locale.KOREA); + + logger.info("GWMain Started."); + + Thread thread = new Thread() { + @Override + public void run() { + Azu azu = new Azu(); + try { + azu.init(); + azu.start(); + azu.join(); + logger.error("Stop ksan azu ..."); + } catch (IllegalStateException e) { + logger.error(e.getMessage()); + System.exit(1); + } catch (Exception e) { + logger.error(e.getMessage()); + System.exit(1); + } + } + }; + thread.start(); + gw = new GW(); try { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/AbortMultipartUpload.java b/core/src/com/pspace/ifs/ksan/gw/api/AbortMultipartUpload.java index dd8d060b..4995b81c 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/AbortMultipartUpload.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/AbortMultipartUpload.java @@ -45,10 +45,6 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -58,6 +54,8 @@ public void process() throws GWException { DataAbortMultipartUpload dataAbortMultipartUpload = new DataAbortMultipartUpload(s3Parameter); dataAbortMultipartUpload.extract(); + checkPolicyBucket(GWConstants.ACTION_ABORT_MULTIPART_UPLOAD, s3Parameter, dataAbortMultipartUpload); + String uploadId = dataAbortMultipartUpload.getUploadId(); s3Parameter.setUploadId(uploadId); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/CompleteMultipartUpload.java b/core/src/com/pspace/ifs/ksan/gw/api/CompleteMultipartUpload.java index 8586e1bf..8dc584e8 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/CompleteMultipartUpload.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/CompleteMultipartUpload.java @@ -44,6 +44,7 @@ import com.pspace.ifs.ksan.libs.multipart.Multipart; import com.pspace.ifs.ksan.libs.multipart.Part; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.objmanager.Metadata; @@ -68,10 +69,6 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -91,10 +88,10 @@ public void process() throws GWException { completeMultipartUpload = xmlMapper.readValue(multipartXml, CompleteMultipartUploadRequest.class); } catch (JsonMappingException e) { PrintStack.logging(logger, e); - new GWException(GWErrorCode.INTERNAL_SERVER_ERROR, s3Parameter); + throw new GWException(GWErrorCode.INTERNAL_SERVER_ERROR, s3Parameter); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); - new GWException(GWErrorCode.INTERNAL_SERVER_ERROR, s3Parameter); + throw new GWException(GWErrorCode.INTERNAL_SERVER_ERROR, s3Parameter); } if(completeMultipartUpload.parts == null || completeMultipartUpload.parts.size() == 0) { @@ -188,7 +185,19 @@ public void process() throws GWException { String versioningStatus = getBucketVersioning(bucket); String versionId = null; Metadata objMeta = null; + try { + // check exist object + objMeta = open(bucket, object); + if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { + versionId = String.valueOf(System.nanoTime()); + } else { + versionId = GWConstants.VERSIONING_DISABLE_TAIL; + } + } catch (GWException e) { + logger.info(e.getMessage()); + // reset error code + s3Parameter.setErrorCode(GWConstants.EMPTY_STRING); if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { versionId = String.valueOf(System.nanoTime()); objMeta = createLocal(multipart.getDiskPoolId(), bucket, object, versionId); @@ -196,17 +205,15 @@ public void process() throws GWException { versionId = GWConstants.VERSIONING_DISABLE_TAIL; objMeta = createLocal(multipart.getDiskPoolId(), bucket, object, versionId); } - } catch (GWException e) { - PrintStack.logging(logger, e); - throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } + s3Parameter.setVersionId(versionId); if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { logger.info(GWConstants.LOG_COMPLETE_MULTIPART_VERSION_ID, versionId); s3Parameter.getResponse().addHeader(GWConstants.X_AMZ_VERSION_ID, versionId); } - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); try (Writer writer = s3Parameter.getResponse().getWriter()) { s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); @@ -287,6 +294,7 @@ public void run() { String jsonmeta = ""; try { + // jsonMapper.setSerializationInclusion(Include.NON_NULL); jsonmeta = jsonMapper.writeValueAsString(s3Metadata); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/CopyObject.java b/core/src/com/pspace/ifs/ksan/gw/api/CopyObject.java index 42ace8be..128299ac 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/CopyObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/CopyObject.java @@ -42,6 +42,7 @@ import com.pspace.ifs.ksan.gw.object.S3ObjectOperation; import com.pspace.ifs.ksan.gw.object.S3ServerSideEncryption; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.objmanager.Metadata; @@ -62,21 +63,15 @@ public void process() throws GWException { initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); - try { - object = URLDecoder.decode(object, GWConstants.CHARSET_UTF_8); + object = URLDecoder.decode(object, Constants.CHARSET_UTF_8); } catch (UnsupportedEncodingException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); @@ -98,15 +93,9 @@ public void process() throws GWException { String copySourceIfNoneMatch = dataCopyObject.getCopySourceIfNoneMatch(); String copySourceIfModifiedSince = dataCopyObject.getCopySourceIfModifiedSince(); String copySourceIfUnmodifiedSince = dataCopyObject.getCopySourceIfUnmodifiedSince(); - // String expires = dataCopyObject.getExpires(); String customerAlgorithm = dataCopyObject.getServerSideEncryptionCustomerAlgorithm(); String customerKey = dataCopyObject.getServerSideEncryptionCustomerKey(); String customerKeyMD5 = dataCopyObject.getServerSideEncryptionCustomerKeyMD5(); - // String context = dataCopyObject.getServerSideEncryptionContext(); - // String bucketKeyEnabled = dataCopyObject.getServerSideEncryptionBucketKeyEnabled(); - String copySourceCustomerAlgorithm = dataCopyObject.getCopySourceServerSideEncryptionCustomerAlgorithm(); - String copySourceCustomerKey = dataCopyObject.getCopySourceServerSideEncryptionCustomerKey(); - String copySourceCustomerKeyMD5 = dataCopyObject.getCopySourceServerSideEncryptionCustomerKeyMD5(); Map userMetadata = dataCopyObject.getUserMetadata(); String storageClass = dataCopyObject.getStorageClass(); @@ -123,7 +112,7 @@ public void process() throws GWException { } try { - copySource = URLDecoder.decode(copySource, GWConstants.CHARSET_UTF_8); + copySource = URLDecoder.decode(copySource, Constants.CHARSET_UTF_8); } catch (UnsupportedEncodingException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); @@ -146,6 +135,13 @@ public void process() throws GWException { String srcVersionId = null; setSrcBucket(srcBucket); + S3Bucket s3SrcBucket = new S3Bucket(); + s3SrcBucket.setBucket(srcBucket); + s3SrcBucket.setUserName(getSrcBucket().getUserName()); + s3SrcBucket.setCors(getSrcBucket().getCors()); + s3SrcBucket.setAccess(getSrcBucket().getAccess()); + s3Parameter.setSrcBucket(s3SrcBucket); + String versioningStatus = getSrcBucket().getVersioning(); if (srcObjectName.contains(GWConstants.SUB_PARAMETER_VERSIONID)) { String[] source = sourcePath[1].split(GWConstants.PARAMETER_BACKSLASH_VERSIONID, 2); @@ -153,8 +149,6 @@ public void process() throws GWException { srcVersionId = source[1].replaceAll(GWConstants.DOUBLE_QUOTE, ""); } - String versioningStatus = getSrcBucket().getVersioning(); - Metadata srcMeta = null; if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { if (!Strings.isNullOrEmpty(srcVersionId) && !GWConstants.VERSIONING_DISABLE_TAIL.equals(srcVersionId)) { @@ -166,29 +160,36 @@ public void process() throws GWException { srcMeta = open(srcBucket, srcObjectName); } srcVersionId = srcMeta.getVersionId(); - srcMeta.setAcl(GWUtils.makeOriginalXml(srcMeta.getAcl(), s3Parameter)); + + logger.info("request src versionId : {}, src meta versionId : {}", srcVersionId, srcMeta.getVersionId()); s3Parameter.setSrcBucketName(srcBucket); s3Parameter.setSrcVersionId(srcVersionId); s3Parameter.setSrcPath(srcObjectName); logger.debug(GWConstants.LOG_SOURCE_INFO, srcBucket, srcObjectName, srcVersionId); - checkGrantObject(s3Parameter.isPublicAccess(), srcMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT, s3Parameter, dataCopyObject)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + checkGrantObject(s3Parameter.isPublicAccess(), srcMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + // get metadata - S3Metadata srcMetadata = null; + // S3Metadata srcMetadata = null; + S3Metadata s3Metadata = null; ObjectMapper objectMapper = new ObjectMapper(); try { - srcMetadata = objectMapper.readValue(srcMeta.getMeta(), S3Metadata.class); + // srcMetadata = objectMapper.readValue(srcMeta.getMeta(), S3Metadata.class); + s3Metadata = objectMapper.readValue(srcMeta.getMeta(), S3Metadata.class); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.INTERNAL_SERVER_ERROR, s3Parameter); } // check source customer-key - if (!Strings.isNullOrEmpty(srcMetadata.getCustomerKey())) { - if (!Strings.isNullOrEmpty(copySourceCustomerKey)) { - if (!srcMetadata.getCustomerKey().equals(copySourceCustomerKey)) { + if (!Strings.isNullOrEmpty(s3Metadata.getCustomerKey())) { + if (!Strings.isNullOrEmpty(customerKey)) { + if (!s3Metadata.getCustomerKey().equals(customerKey)) { logger.warn(GWConstants.LOG_COPY_OBJECT_SOURCE_CUSTOMER_KEY_NO_MATCH); throw new GWException(GWErrorCode.KEY_DOES_NOT_MATCH, s3Parameter); } @@ -198,11 +199,11 @@ public void process() throws GWException { } } - S3ObjectEncryption srcObjectEncryption = new S3ObjectEncryption(s3Parameter, srcMetadata); - srcObjectEncryption.build(); + S3ObjectEncryption srcEncryption = new S3ObjectEncryption(s3Parameter, s3Metadata); + srcEncryption.build(); // check - S3Metadata s3Metadata = new S3Metadata(); + // S3Metadata s3Metadata = new S3Metadata(); s3Metadata.setCustomerKey(customerKey); s3Metadata.setCustomerAlgorithm(customerAlgorithm); s3Metadata.setCustomerKeyMD5(customerKeyMD5); @@ -210,7 +211,7 @@ public void process() throws GWException { String bucketEncryption = getBucketInfo().getEncryption(); logger.debug(GWConstants.LOG_COPY_OBJECT_ENCRYPTION, bucketEncryption); // check encryption - S3ServerSideEncryption encryption = new S3ServerSideEncryption(bucketEncryption, s3Metadata, s3Parameter); + S3ServerSideEncryption encryption = new S3ServerSideEncryption(bucketEncryption, serversideEncryption, customerAlgorithm, customerKey, customerKeyMD5, s3Parameter); encryption.build(); if (encryption.isEncryptionEnabled()) { s3Metadata.setServersideEncryption(GWConstants.AES256); @@ -220,15 +221,17 @@ public void process() throws GWException { // check match if (!Strings.isNullOrEmpty(copySourceIfMatch)) { - logger.debug(GWConstants.LOG_SOURCE_ETAG_MATCH, srcMetadata.getETag(), copySourceIfMatch.replace(GWConstants.DOUBLE_QUOTE, "")); - if (!GWUtils.maybeQuoteETag(srcMetadata.getETag()).equals(copySourceIfMatch.replace(GWConstants.DOUBLE_QUOTE, ""))) { + if (!GWUtils.maybeQuoteETag(s3Metadata.getETag()).equals(GWUtils.maybeQuoteETag(copySourceIfMatch))) { + logger.debug(GWConstants.LOG_SOURCE_ETAG_MATCH, s3Metadata.getETag(), copySourceIfMatch); + logger.info(GWErrorCode.PRECONDITION_FAILED.getMessage()); throw new GWException(GWErrorCode.PRECONDITION_FAILED, s3Parameter); } } if (!Strings.isNullOrEmpty(copySourceIfNoneMatch)) { - logger.debug(GWConstants.LOG_SOURCE_ETAG_MATCH, srcMetadata.getETag(), copySourceIfNoneMatch.replace(GWConstants.DOUBLE_QUOTE, "")); - if (GWUtils.maybeQuoteETag(srcMetadata.getETag()).equals(copySourceIfNoneMatch.replace(GWConstants.DOUBLE_QUOTE, ""))) { + if (GWUtils.maybeQuoteETag(s3Metadata.getETag()).equals(GWUtils.maybeQuoteETag(copySourceIfNoneMatch))) { + logger.debug(GWConstants.LOG_SOURCE_ETAG_MATCH, s3Metadata.getETag(), copySourceIfNoneMatch); + logger.info(GWErrorCode.PRECONDITION_FAILED.getMessage()); throw new GWException(GWErrorCode.DOES_NOT_MATCH, String.format(GWConstants.LOG_ETAG_IS_MISMATCH), s3Parameter); } } @@ -237,8 +240,10 @@ public void process() throws GWException { long copySourceIfModifiedSinceLong = Long.parseLong(copySourceIfModifiedSince); if (copySourceIfModifiedSinceLong != -1) { Date modifiedSince = new Date(copySourceIfModifiedSinceLong); - if (srcMetadata.getLastModified().before(modifiedSince)) { - throw new GWException(GWErrorCode.DOES_NOT_MATCH, String.format(GWConstants.LOG_MATCH_BEFORE, srcMetadata.getLastModified(), modifiedSince), s3Parameter); + if (s3Metadata.getLastModified().before(modifiedSince)) { + logger.info(GWErrorCode.DOES_NOT_MATCH.getMessage() + + String.format(" : %1$s is before %2$s", s3Metadata.getLastModified(), modifiedSince)); + throw new GWException(GWErrorCode.DOES_NOT_MATCH, String.format(GWConstants.LOG_MATCH_BEFORE, s3Metadata.getLastModified(), modifiedSince), s3Parameter); } } } @@ -247,8 +252,10 @@ public void process() throws GWException { long copySourceIfUnmodifiedSinceLong = Long.parseLong(copySourceIfUnmodifiedSince); if (copySourceIfUnmodifiedSinceLong != -1) { Date unmodifiedSince = new Date(copySourceIfUnmodifiedSinceLong); - if (srcMetadata.getLastModified().after(unmodifiedSince)) { - throw new GWException(GWErrorCode.PRECONDITION_FAILED, String.format(GWConstants.LOG_MATCH_AFTER, srcMetadata.getLastModified(), unmodifiedSince), s3Parameter); + if (s3Metadata.getLastModified().after(unmodifiedSince)) { + logger.info(GWErrorCode.PRECONDITION_FAILED.getMessage() + + String.format(" : %1$s is after %2$s", s3Metadata.getLastModified(), unmodifiedSince)); + throw new GWException(GWErrorCode.PRECONDITION_FAILED, String.format(GWConstants.LOG_MATCH_AFTER, s3Metadata.getLastModified(), unmodifiedSince), s3Parameter); } } } @@ -273,16 +280,14 @@ public void process() throws GWException { dataCopyObject.getGrantFullControl(), dataCopyObject.getGrantReadAcp(), dataCopyObject.getGrantWriteAcp(), - s3Parameter); + s3Parameter, + false); // check replace or copy boolean bReplaceMetadata = false; logger.debug(GWConstants.LOG_COPY_OBJECT_METADATA_DIRECTIVE, metadataDirective); if (!Strings.isNullOrEmpty(metadataDirective) && metadataDirective.equalsIgnoreCase(GWConstants.REPLACE)) { bReplaceMetadata = true; - } else { - s3Metadata.setUserMetadataMap(srcMetadata.getUserMetadataMap()); - s3Metadata.setContentType(srcMetadata.getContentType()); } s3Metadata.setOwnerId(s3Parameter.getUser().getUserId()); @@ -351,49 +356,55 @@ public void process() throws GWException { String jsonmeta = ""; // update if (s3Parameter.getSrcBucketName().equals(bucket) && s3Parameter.getSrcPath().equals(object)) { - if (!Strings.isNullOrEmpty(metadataDirective) && bReplaceMetadata) { + if (!Strings.isNullOrEmpty(metadataDirective)) { // update metadata - try { - S3Metadata metaClass = objectMapper.readValue(srcMeta.getMeta(), S3Metadata.class); - s3Metadata.setTier(GWConstants.AWS_TIER_STANTARD); - s3Metadata.setContentLength(metaClass.getContentLength()); - s3Metadata.setETag(metaClass.getETag()); - } catch (JsonProcessingException e) { - PrintStack.logging(logger, e); - throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + if (bReplaceMetadata) { + try { + S3Metadata metaClass = objectMapper.readValue(srcMeta.getMeta(), S3Metadata.class); + s3Metadata.setTier(GWConstants.AWS_TIER_STANTARD); + s3Metadata.setContentLength(metaClass.getContentLength()); + s3Metadata.setETag(metaClass.getETag()); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + s3Metadata.setLastModified(new Date()); + try { + jsonmeta = objectMapper.writeValueAsString(s3Metadata); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + logger.debug(GWConstants.LOG_COPY_OBJECT_META, jsonmeta); + srcMeta.setMeta(jsonmeta); + updateObjectMeta(srcMeta); + + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); + XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); + try (Writer writer = s3Parameter.getResponse().getWriter()) { + s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); + XMLStreamWriter xmlout = xmlOutputFactory.createXMLStreamWriter(writer); + xmlout.writeStartDocument(); + xmlout.writeStartElement(GWConstants.COPY_OBJECT_RESULT); + xmlout.writeDefaultNamespace(GWConstants.AWS_XMLNS); + + writeSimpleElement(xmlout, GWConstants.LAST_MODIFIED, formatDate(s3Metadata.getLastModified())); + writeSimpleElement(xmlout, GWConstants.ETAG, GWUtils.maybeQuoteETag(s3Metadata.getETag())); + + xmlout.writeEndElement(); + xmlout.flush(); + } catch (XMLStreamException | IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + return; + } else { + logger.error(GWErrorCode.INVALID_REQUEST.getMessage()); + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); } - s3Metadata.setLastModified(new Date()); - try { - jsonmeta = objectMapper.writeValueAsString(s3Metadata); - } catch (JsonProcessingException e) { - PrintStack.logging(logger, e); - throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); - } - logger.debug(GWConstants.LOG_COPY_OBJECT_META, jsonmeta); - srcMeta.setMeta(jsonmeta); - updateObjectMeta(srcMeta); - - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); - XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); - try (Writer writer = s3Parameter.getResponse().getWriter()) { - s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); - XMLStreamWriter xmlout = xmlOutputFactory.createXMLStreamWriter(writer); - xmlout.writeStartDocument(); - xmlout.writeStartElement(GWConstants.COPY_OBJECT_RESULT); - xmlout.writeDefaultNamespace(GWConstants.AWS_XMLNS); - - writeSimpleElement(xmlout, GWConstants.LAST_MODIFIED, formatDate(s3Metadata.getLastModified())); - writeSimpleElement(xmlout, GWConstants.ETAG, GWUtils.maybeQuoteETag(s3Metadata.getETag())); - - xmlout.writeEndElement(); - xmlout.flush(); - } catch (XMLStreamException | IOException e) { - PrintStack.logging(logger, e); - throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); - } - - s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); - return; + } else { logger.error(GWErrorCode.INVALID_REQUEST.getMessage()); throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); @@ -403,21 +414,36 @@ public void process() throws GWException { versioningStatus = getBucketVersioning(bucket); String versionId = null; - if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { - versionId = String.valueOf(System.nanoTime()); - } else { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; + Metadata objMeta = null; + try { + // check exist object + objMeta = open(bucket, object); + if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { + versionId = String.valueOf(System.nanoTime()); + } else { + versionId = GWConstants.VERSIONING_DISABLE_TAIL; + } + } catch (GWException e) { + logger.info(e.getMessage()); + // reset error code + s3Parameter.setErrorCode(GWConstants.EMPTY_STRING); + if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { + versionId = String.valueOf(System.nanoTime()); + objMeta = createLocal(diskpoolId, bucket, object, versionId); + } else { + versionId = GWConstants.VERSIONING_DISABLE_TAIL; + objMeta = createLocal(diskpoolId, bucket, object, versionId); + } } - Metadata objMeta = createLocal(diskpoolId, bucket, object, versionId); - // Metadata objMeta = createCopy(srcBucket, srcObjectName, srcVersionId, bucket, object); + s3Parameter.setVersionId(versionId); S3ObjectOperation objectOperation = new S3ObjectOperation(objMeta, s3Metadata, s3Parameter, versionId, encryption); - S3Object s3Object = objectOperation.copyObject(srcMeta, srcObjectEncryption); + S3Object s3Object = objectOperation.copyObject(srcMeta, srcEncryption); s3Metadata.setETag(s3Object.getEtag()); s3Metadata.setContentLength(s3Object.getFileSize()); - s3Metadata.setTier(GWConstants.AWS_TIER_STANTARD); + s3Metadata.setTier(storageClass); s3Metadata.setLastModified(s3Object.getLastModified()); s3Metadata.setDeleteMarker(s3Object.getDeleteMarker()); s3Metadata.setVersionId(s3Object.getVersionId()); @@ -444,7 +470,7 @@ public void process() throws GWException { throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); try (Writer writer = s3Parameter.getResponse().getWriter()) { s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/CreateBucket.java b/core/src/com/pspace/ifs/ksan/gw/api/CreateBucket.java index 2bfe2e22..12f71fb2 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/CreateBucket.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/CreateBucket.java @@ -28,6 +28,8 @@ import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.objmanager.Bucket; import com.pspace.ifs.ksan.libs.DiskManager; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.libs.PrintStack; import org.slf4j.LoggerFactory; @@ -79,10 +81,12 @@ public void process() throws GWException { dataCreateBucket.getGrantFullControl(), dataCreateBucket.getGrantReadAcp(), dataCreateBucket.getGrantWriteAcp(), - s3Parameter); + s3Parameter, + false); logger.debug(GWConstants.LOG_ACL, xml); int result = 0; + logger.info("user : {}, {}, {}", s3Parameter.getUser().getAccessKey(), s3Parameter.getUser().getUserDiskpoolId(GWConstants.AWS_TIER_STANTARD), s3Parameter.getUser().getUserDefaultDiskpoolId()); String diskpoolId = s3Parameter.getUser().getUserDefaultDiskpoolId(); logger.info("user default diskpoolId : {}", diskpoolId); @@ -93,6 +97,14 @@ public void process() throws GWException { bucket.setAcl(xml); bucket.setDiskPoolId(diskpoolId); + S3Bucket s3Bucket = new S3Bucket(); + s3Bucket.setBucket(bucket.getName()); + s3Bucket.setUserName(bucket.getUserName()); + // s3Bucket.setPolicy(getBucketInfo().getPolicy()); + // s3Bucket.setCors(getBucketInfo().getCors()); + // s3Bucket.setAccess(getBucketInfo().getAccess()); + s3Parameter.setBucket(s3Bucket); + if (!Strings.isNullOrEmpty(dataCreateBucket.getBucketObjectLockEnabled()) && GWConstants.STRING_TRUE.equalsIgnoreCase(dataCreateBucket.getBucketObjectLockEnabled())) { logger.info(GWConstants.LOG_CREATE_BUCKET_VERSIONING_ENABLED_OBJECT_LOCK_TRUE); String objectLockXml = GWConstants.OBJECT_LOCK_XML; diff --git a/core/src/com/pspace/ifs/ksan/gw/api/CreateMultipartUpload.java b/core/src/com/pspace/ifs/ksan/gw/api/CreateMultipartUpload.java index 82db8570..aaef30ab 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/CreateMultipartUpload.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/CreateMultipartUpload.java @@ -20,6 +20,7 @@ import javax.xml.stream.XMLStreamWriter; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.google.common.base.Strings; import com.pspace.ifs.ksan.gw.data.DataCreateMultipartUpload; @@ -56,21 +57,19 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); - DataCreateMultipartUpload dataCreateMultipartUpload = new DataCreateMultipartUpload(s3Parameter); dataCreateMultipartUpload.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT, s3Parameter, dataCreateMultipartUpload)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + accessControlPolicy = new AccessControlPolicy(); accessControlPolicy.aclList = new AccessControlList(); accessControlPolicy.aclList.grants = new ArrayList(); @@ -91,7 +90,8 @@ public void process() throws GWException { dataCreateMultipartUpload.getGrantFullControl(), dataCreateMultipartUpload.getGrantReadAcp(), dataCreateMultipartUpload.getGrantWriteAcp(), - s3Parameter); + s3Parameter, + false); String customerAlgorithm = dataCreateMultipartUpload.getServerSideEncryptionCustomerAlgorithm(); String customerKey = dataCreateMultipartUpload.getServerSideEncryptionCustomerKey(); @@ -169,6 +169,7 @@ public void process() throws GWException { ObjectMapper jsonMapper = new ObjectMapper(); String metaJson = ""; try { + // jsonMapper.setSerializationInclusion(Include.NON_NULL); metaJson = jsonMapper.writeValueAsString(s3Metadata); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucket.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucket.java index 449a6de9..174f1f9b 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucket.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucket.java @@ -13,6 +13,7 @@ import jakarta.servlet.http.HttpServletResponse; +import com.pspace.ifs.ksan.gw.data.DataDeleteBucket; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -21,8 +22,7 @@ import com.pspace.ifs.ksan.gw.utils.GWUtils; import org.slf4j.LoggerFactory; - - +import com.google.common.base.Strings; public class DeleteBucket extends S3Request{ @@ -36,13 +36,19 @@ public void process() throws GWException { logger.info(GWConstants.LOG_DELETE_BUCKET_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); + DataDeleteBucket dataDeleteBucket = new DataDeleteBucket(s3Parameter); + dataDeleteBucket.extract(); + + String expectedBucketOwner = dataDeleteBucket.getExpectedBucketOwner(); + if (!Strings.isNullOrEmpty(expectedBucketOwner)) { + if (!isBucketOwner(expectedBucketOwner)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketCors.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketCors.java index 204e36e3..63658ad6 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketCors.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketCors.java @@ -33,10 +33,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketEncryption.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketEncryption.java index 1264d8c6..0ac88932 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketEncryption.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketEncryption.java @@ -34,10 +34,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketLifeCycle.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketLifeCycle.java index 3b8c85b8..2a07169b 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketLifeCycle.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketLifeCycle.java @@ -33,10 +33,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketLogging.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketLogging.java new file mode 100644 index 00000000..1dd2b059 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketLogging.java @@ -0,0 +1,48 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import jakarta.servlet.http.HttpServletResponse; + +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; + +import org.slf4j.LoggerFactory; + +public class DeleteBucketLogging extends S3Request { + + public DeleteBucketLogging(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(DeleteBucketLogging.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_DELETE_BUCKET_OBJECT_LOCK_START); + + String bucket = s3Parameter.getBucketName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + updateBucketLogging(bucket, ""); + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_NO_CONTENT); + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketObjectLock.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketObjectLock.java index e62c0114..60420dff 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketObjectLock.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketObjectLock.java @@ -34,10 +34,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketPolicy.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketPolicy.java index bf34df87..56c3b2d7 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketPolicy.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketPolicy.java @@ -18,6 +18,7 @@ import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.gw.data.DataPutBucketPolicy; import org.slf4j.LoggerFactory; @@ -32,15 +33,17 @@ public void process() throws GWException { logger.info(GWConstants.LOG_DELETE_BUCKET_POLICY_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } + + DataPutBucketPolicy dataPutBucketPolicy = new DataPutBucketPolicy(s3Parameter); + dataPutBucketPolicy.extract(); + + checkPolicyBucket(GWConstants.ACTION_DELETE_BUCKET_POLICY, s3Parameter, dataPutBucketPolicy); updateBucketPolicy(bucket, ""); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketReplication.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketReplication.java index 7695c41c..e6107472 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketReplication.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketReplication.java @@ -33,10 +33,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketTagIndex.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketTagIndex.java new file mode 100644 index 00000000..61b500e3 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketTagIndex.java @@ -0,0 +1,47 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import jakarta.servlet.http.HttpServletResponse; + +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; + +import org.slf4j.LoggerFactory; + +public class DeleteBucketTagIndex extends S3Request { + public DeleteBucketTagIndex(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(DeleteBucketTagIndex.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_DELETE_BUCKET_TAG_INDEX_START); + + String bucket = s3Parameter.getBucketName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + updateBucketTagIndex(bucket, false); + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_NO_CONTENT); + } +} \ No newline at end of file diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketTagging.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketTagging.java index 545acc96..96bc08a4 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketTagging.java @@ -33,10 +33,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketWebsite.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketWebsite.java index fc0f90be..6e87ee7c 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketWebsite.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteBucketWebsite.java @@ -18,6 +18,7 @@ import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.gw.data.DataDeleteBucketWebsite; import org.slf4j.LoggerFactory; @@ -33,16 +34,18 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } + DataDeleteBucketWebsite dataDeleteBucketWebsite = new DataDeleteBucketWebsite(s3Parameter); + dataDeleteBucketWebsite.extract(); + + checkPolicyBucket(GWConstants.ACTION_DELETE_BUCKET_WEBSITE, s3Parameter, dataDeleteBucketWebsite); + updateBucketWeb(bucket, ""); s3Parameter.getResponse().setStatus(HttpServletResponse.SC_NO_CONTENT); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteObject.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteObject.java index 0d206644..3803aec7 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteObject.java @@ -15,6 +15,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.google.common.base.Strings; import com.pspace.ifs.ksan.gw.data.DataDeleteObject; @@ -49,17 +50,11 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); logger.debug(GWConstants.LOG_DELETE_OBJECT, bucket, object); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); DataDeleteObject dataDeleteObject = new DataDeleteObject(s3Parameter); dataDeleteObject.extract(); @@ -70,11 +65,23 @@ public void process() throws GWException { s3Metadata.setOwnerName(s3Parameter.getUser().getUserName()); String versionId = dataDeleteObject.getVersionId(); + s3Parameter.setVersionId(versionId); + logger.debug("with version id : {}", versionId); + + if (Strings.isNullOrEmpty(versionId)) { + if (!checkPolicyBucket(GWConstants.ACTION_DELETE_OBJECT, s3Parameter, dataDeleteObject)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } else { + if (!checkPolicyBucket(GWConstants.ACTION_DELETE_OBJECT_VERSION, s3Parameter, dataDeleteObject)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } + boolean isLastVersion = true; String deleteMarker = null; String versioningStatus = null; - versioningStatus = getBucketVersioning(bucket); Metadata objMeta = null; @@ -87,7 +94,7 @@ public void process() throws GWException { } catch (GWException e) { if (e.getError().equals(GWErrorCode.NO_SUCH_KEY) && Strings.isNullOrEmpty(versionId) && versioningStatus.equalsIgnoreCase(GWConstants.VERSIONING_ENABLED)) { objMeta = createLocal(bucket, object); - putDeleteMarker(bucket, object, s3Metadata, objMeta); + putDeleteMarker(bucket, object, GWConstants.VERSIONING_DISABLE_TAIL, s3Metadata, objMeta); } s3Parameter.getResponse().addHeader(GWConstants.X_AMZ_VERSION_ID, versionId); s3Parameter.getResponse().setStatus(HttpServletResponse.SC_NO_CONTENT); @@ -109,12 +116,14 @@ public void process() throws GWException { } else { if (versioningStatus.equalsIgnoreCase(GWConstants.VERSIONING_ENABLED)) { // Bucket Versioning Enabled logger.debug(GWConstants.LOG_DELETE_OBJECT_BUCKET_VERSIONING_ENABLED); + retentionCheck(objMeta.getMeta(), dataDeleteObject.getBypassGovernanceRetention(), s3Parameter); + if (Strings.isNullOrEmpty(versionId)) { // request versionId is null if (deleteMarker.equalsIgnoreCase(GWConstants.OBJECT_TYPE_MARK)) { remove(bucket, object, GWConstants.VERSIONING_DISABLE_TAIL); } else { // put delete marker - putDeleteMarker(bucket, object, s3Metadata, objMeta); + putDeleteMarker(bucket, object, String.valueOf(System.nanoTime()), s3Metadata, objMeta); } } else { // request with versionId if (isLastVersion) { @@ -134,9 +143,8 @@ public void process() throws GWException { if (deleteMarker.equalsIgnoreCase(GWConstants.OBJECT_TYPE_MARK)) { remove(bucket, object, GWConstants.OBJECT_TYPE_MARK); } else { - remove(bucket, object, objMeta.getVersionId()); - objectOperation.deleteObject(); - logger.info("delete object - bucket : {}, object : {}, versionId : {}", bucket, object, versionId); + // put delete marker + putDeleteMarker(bucket, object, GWConstants.VERSIONING_DISABLE_TAIL, s3Metadata, objMeta); } } else { remove(bucket, object, objMeta.getVersionId()); @@ -157,9 +165,8 @@ public void process() throws GWException { s3Parameter.getResponse().setStatus(HttpServletResponse.SC_NO_CONTENT); } - private void putDeleteMarker(String bucket, String object, S3Metadata s3Metadata, Metadata objMeta) throws GWException { + private void putDeleteMarker(String bucket, String object, String versionId, S3Metadata s3Metadata, Metadata objMeta) throws GWException { try { - String versionId = String.valueOf(System.nanoTime()); s3Metadata.setDeleteMarker(GWConstants.OBJECT_TYPE_MARK); s3Metadata.setLastModified(new Date()); s3Metadata.setContentLength(0L); @@ -167,6 +174,7 @@ private void putDeleteMarker(String bucket, String object, S3Metadata s3Metadata ObjectMapper jsonMapper = new ObjectMapper(); String jsonmeta = ""; + // jsonMapper.setSerializationInclusion(Include.NON_NULL); jsonmeta = jsonMapper.writeValueAsString(s3Metadata); int result; objMeta.set("", "", jsonmeta, "", 0L); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteObjectTagging.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteObjectTagging.java index 605cba3e..c734f8ee 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteObjectTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteObjectTagging.java @@ -37,11 +37,7 @@ public void process() throws GWException { initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -52,17 +48,24 @@ public void process() throws GWException { dataDeleteObjectTagging.extract(); String versionId = dataDeleteObjectTagging.getVersionId(); - + s3Parameter.setVersionId(versionId); + Metadata objMeta = null; if (Strings.isNullOrEmpty(versionId)) { objMeta = open(bucket, object); - versionId = objMeta.getVersionId(); } else { objMeta = open(bucket, object, versionId); } - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); - checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + if (Strings.isNullOrEmpty(versionId)) { + if (!checkPolicyBucket(GWConstants.ACTION_DELETE_OBJECT_TAGGING, s3Parameter, dataDeleteObjectTagging)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } else { + if (!checkPolicyBucket(GWConstants.ACTION_DELETE_OBJECT_VERSION_TAGGING, s3Parameter, dataDeleteObjectTagging)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } objMeta.setTag(""); updateObjectTagging(objMeta); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeleteObjects.java b/core/src/com/pspace/ifs/ksan/gw/api/DeleteObjects.java index 962536d7..51d1cbe9 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeleteObjects.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeleteObjects.java @@ -10,6 +10,8 @@ */ package com.pspace.ifs.ksan.gw.api; +import java.util.Date; + import java.io.IOException; import java.io.Writer; import java.security.InvalidParameterException; @@ -24,6 +26,7 @@ import com.fasterxml.jackson.core.JacksonException; import com.fasterxml.jackson.core.JsonParseException; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.dataformat.xml.XmlMapper; import com.google.common.base.Strings; @@ -57,17 +60,11 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); DataDeleteObjects dataDeleteObjects = new DataDeleteObjects(s3Parameter); dataDeleteObjects.extract(); @@ -102,7 +99,7 @@ public void process() throws GWException { logger.debug(GWConstants.LOG_DELETE_OBJECTS_SIZE, objectNames.size()); for (Objects object : objectNames) { - deleteObject(s3Parameter, object.objectName, object.versionId, xmlStreamWriter, deleteMultipleObjectsRequest.quiet); + deleteObject(s3Parameter, object.objectName, object.versionId, xmlStreamWriter, deleteMultipleObjectsRequest.quiet, dataDeleteObjects); xmlStreamWriter.flush(); // In Tomcat, if you use flush(), you lose connection. jakarta, need to check } @@ -125,7 +122,7 @@ public void process() throws GWException { s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); } - public void deleteObject(S3Parameter s3Parameter, String object, String versionId, XMLStreamWriter xml, boolean quiet) throws GWException { + public void deleteObject(S3Parameter s3Parameter, String object, String versionId, XMLStreamWriter xml, boolean quiet, DataDeleteObjects dataDeleteObjects) throws GWException { String bucket = s3Parameter.getBucketName(); String versioningStatus = getBucketInfo().getVersioning(); @@ -164,6 +161,16 @@ public void deleteObject(S3Parameter s3Parameter, String object, String versionI S3ObjectOperation objectOperation = new S3ObjectOperation(objMeta, null, s3Parameter, versionId, null); try { + if (Strings.isNullOrEmpty(versionId)) { + if (!checkPolicyBucket(GWConstants.ACTION_DELETE_OBJECT, s3Parameter, dataDeleteObjects)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } else { + if (!checkPolicyBucket(GWConstants.ACTION_DELETE_OBJECT_VERSION, s3Parameter, dataDeleteObjects)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } + if (versioningStatus.equalsIgnoreCase(GWConstants.VERSIONING_ENABLED)) { // Bucket Versioning Enabled logger.debug(GWConstants.LOG_DELETE_OBJECT_BUCKET_VERSIONING_ENABLED); if (GWConstants.VERSIONING_DISABLE_TAIL.equals(versionId)) { // request versionId is null @@ -178,13 +185,18 @@ public void deleteObject(S3Parameter s3Parameter, String object, String versionI try { S3Metadata s3Metadata = new S3Metadata(); s3Metadata.setName(object); + s3Metadata.setLastModified(new Date()); + s3Metadata.setContentLength(0L); + s3Metadata.setTier(GWConstants.AWS_TIER_STANTARD); s3Metadata.setOwnerId(s3Parameter.getUser().getUserId()); s3Metadata.setOwnerName(s3Parameter.getUser().getUserName()); s3Metadata.setDeleteMarker(GWConstants.OBJECT_TYPE_MARK); + versionId = String.valueOf(System.nanoTime()); s3Metadata.setVersionId(versionId); ObjectMapper jsonMapper = new ObjectMapper(); String jsonmeta = ""; + // jsonMapper.setSerializationInclusion(Include.NON_NULL); jsonmeta = jsonMapper.writeValueAsString(s3Metadata); int result; objMeta.set("", "", jsonmeta, "", 0L); @@ -231,13 +243,18 @@ public void deleteObject(S3Parameter s3Parameter, String object, String versionI try { S3Metadata s3Metadata = new S3Metadata(); s3Metadata.setName(object); + s3Metadata.setLastModified(new Date()); + s3Metadata.setContentLength(0L); + s3Metadata.setTier(GWConstants.AWS_TIER_STANTARD); s3Metadata.setOwnerId(s3Parameter.getUser().getUserId()); s3Metadata.setOwnerName(s3Parameter.getUser().getUserName()); s3Metadata.setDeleteMarker(GWConstants.OBJECT_TYPE_MARK); + versionId = GWConstants.VERSIONING_DISABLE_TAIL; s3Metadata.setVersionId(versionId); ObjectMapper jsonMapper = new ObjectMapper(); String jsonmeta = ""; + // jsonMapper.setSerializationInclusion(Include.NON_NULL); jsonmeta = jsonMapper.writeValueAsString(s3Metadata); int result; diff --git a/core/src/com/pspace/ifs/ksan/gw/api/DeletePublicAccessBlock.java b/core/src/com/pspace/ifs/ksan/gw/api/DeletePublicAccessBlock.java index dea47c3a..0b459666 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/DeletePublicAccessBlock.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/DeletePublicAccessBlock.java @@ -33,10 +33,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketAcl.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketAcl.java index d9beaa98..d800734f 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketAcl.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketAcl.java @@ -39,10 +39,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -52,14 +49,16 @@ public void process() throws GWException { DataGetObjectAcl dataGetObjectAcl = new DataGetObjectAcl(s3Parameter); dataGetObjectAcl.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_GET_BUCKET_ACL, s3Parameter, dataGetObjectAcl)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } + String aclInfo = getBucketInfo().getAcl(); logger.debug(GWConstants.LOG_ACL, aclInfo); if (!aclInfo.contains(GWConstants.XML_VERSION)) { aclInfo = GWConstants.XML_VERSION_FULL_STANDALONE + aclInfo; } - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); - try { if (!Strings.isNullOrEmpty(aclInfo)) { s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketCors.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketCors.java index a5d680e8..940cf51c 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketCors.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketCors.java @@ -15,6 +15,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketCors; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -37,17 +38,19 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + DataGetBucketCors dataGetBucketCors = new DataGetBucketCors(s3Parameter); + dataGetBucketCors.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_GET_BUCKET_CORS, s3Parameter, dataGetBucketCors)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } String cors = getBucketInfo().getCors(); logger.debug(GWConstants.LOG_GET_BUCKET_CORS, cors); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketEncryption.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketEncryption.java index 32d7c2e8..1c57f687 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketEncryption.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketEncryption.java @@ -15,6 +15,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketEncryption; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -38,17 +39,19 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + DataGetBucketEncryption dataGetBucketEncryption = new DataGetBucketEncryption(s3Parameter); + dataGetBucketEncryption.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_GET_ENCRYPTION_CONFIGURATION, s3Parameter, dataGetBucketEncryption)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } String encryption = getBucketInfo().getEncryption(); logger.debug(GWConstants.LOG_GET_BUCKET_ENCRYPTION, encryption); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLifecycleConfiguration.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLifecycleConfiguration.java index 4b16962a..def811c1 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLifecycleConfiguration.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLifecycleConfiguration.java @@ -15,6 +15,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketLifecycleConfiguration; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -37,17 +38,19 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } + + DataGetBucketLifecycleConfiguration dataGetBucketLifecycleConfiguration = new DataGetBucketLifecycleConfiguration(s3Parameter); + dataGetBucketLifecycleConfiguration.extract(); - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + if (!checkPolicyBucket(GWConstants.ACTION_GET_LIFECYCLE_CONFIGURATION, s3Parameter, dataGetBucketLifecycleConfiguration)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } String lifecycle = getBucketInfo().getLifecycle(); logger.debug(GWConstants.LOG_GET_BUCKET_LIFECYCLE, lifecycle); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLocation.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLocation.java index 47a12412..e84e45ae 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLocation.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLocation.java @@ -39,10 +39,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLogging.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLogging.java new file mode 100644 index 00000000..6b81cb35 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketLogging.java @@ -0,0 +1,72 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import java.io.IOException; + +import jakarta.servlet.http.HttpServletResponse; +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketLogging; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; + +import org.slf4j.LoggerFactory; + +public class GetBucketLogging extends S3Request { + + public GetBucketLogging(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(GetBucketLogging.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_GET_BUCKET_LOGGING_START); + String bucket = s3Parameter.getBucketName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + DataGetBucketLogging dataGetBucketLogging = new DataGetBucketLogging(s3Parameter); + dataGetBucketLogging.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_GET_BUCKET_LOGGING, s3Parameter, dataGetBucketLogging)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } + + String logging = getBucketInfo().getLogging(); + logger.debug(GWConstants.LOG_GET_BUCKET_LOGGING, logging); + + try { + if (!Strings.isNullOrEmpty(logging)) { + s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); + s3Parameter.getResponse().getOutputStream().write(logging.getBytes()); + } else { + s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); + s3Parameter.getResponse().getOutputStream().write(GWConstants.LOG_GET_BUCKET_LOGGING_EMPTY.getBytes()); + } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetObjectLockConfiguration.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketObjectLock.java similarity index 59% rename from core/src/com/pspace/ifs/ksan/gw/api/GetObjectLockConfiguration.java rename to core/src/com/pspace/ifs/ksan/gw/api/GetBucketObjectLock.java index c923c343..ddec4d62 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetObjectLockConfiguration.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketObjectLock.java @@ -14,8 +14,9 @@ import jakarta.servlet.http.HttpServletResponse; -import com.fasterxml.jackson.dataformat.xml.XmlMapper; import com.google.common.base.Strings; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.pspace.ifs.ksan.gw.data.DataGetBucketObjectLock; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.format.ObjectLockConfiguration; @@ -27,53 +28,52 @@ import org.slf4j.LoggerFactory; -public class GetObjectLockConfiguration extends S3Request { +public class GetBucketObjectLock extends S3Request { - public GetObjectLockConfiguration(S3Parameter s3Parameter) { + public GetBucketObjectLock(S3Parameter s3Parameter) { super(s3Parameter); - logger = LoggerFactory.getLogger(GetObjectLockConfiguration.class); + logger = LoggerFactory.getLogger(GetBucketObjectLock.class); } - + @Override public void process() throws GWException { logger.info(GWConstants.LOG_GET_BUCKET_OBJECT_LOCK_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); - GWUtils.checkCors(s3Parameter); - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); - String objectLock = getBucketInfo().getObjectLock(); - logger.debug(GWConstants.LOG_OBJECT_LOCK, objectLock); - if (Strings.isNullOrEmpty(objectLock)) { - throw new GWException(GWErrorCode.OBJECT_LOCK_CONFIGURATION_NOT_FOUND_ERROR, s3Parameter); + DataGetBucketObjectLock dataGetBucketObjectLock = new DataGetBucketObjectLock(s3Parameter); + dataGetBucketObjectLock.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_GET_BUCKET_OBJECT_LOCK_CONFIGURATION, s3Parameter, dataGetBucketObjectLock)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); } + + String objectLock = getBucketInfo().getObjectLock(); + logger.debug(GWConstants.LOG_OBJECT_LOCK, objectLock); + + if (Strings.isNullOrEmpty(objectLock)) { + throw new GWException(GWErrorCode.OBJECT_LOCK_CONFIGURATION_NOT_FOUND_ERROR, s3Parameter); + } - try { - ObjectLockConfiguration configuration = new XmlMapper().readValue(objectLock, ObjectLockConfiguration.class); - if (!GWConstants.STATUS_ENABLED.equals(configuration.objectLockEnabled)) { + try { + ObjectLockConfiguration oc = new XmlMapper().readValue(objectLock, ObjectLockConfiguration.class); + if (!oc.objectLockEnabled.equals(GWConstants.STATUS_ENABLED)) { throw new GWException(GWErrorCode.OBJECT_LOCK_CONFIGURATION_NOT_FOUND_ERROR, s3Parameter); } - - if (!Strings.isNullOrEmpty(objectLock)) { - s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); - s3Parameter.getResponse().getOutputStream().write(objectLock.getBytes()); - } + s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); + s3Parameter.getResponse().getOutputStream().write(objectLock.getBytes()); } catch (IOException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } - - s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); } - } diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketPolicy.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketPolicy.java index 965f2756..28cb32e4 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketPolicy.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketPolicy.java @@ -14,6 +14,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketPolicy; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -35,17 +36,19 @@ public void process() throws GWException { logger.info(GWConstants.LOG_GET_BUCKET_POLICY_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + DataGetBucketPolicy dataGetBucketPolicy = new DataGetBucketPolicy(s3Parameter); + dataGetBucketPolicy.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_GET_BUCKET_POLICY, s3Parameter, dataGetBucketPolicy)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } String policy = getBucketInfo().getPolicy(); logger.debug(GWConstants.LOG_GET_BUCKET_POLICY, policy); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketPolicyStatus.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketPolicyStatus.java index aa1ef6a5..691793c6 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketPolicyStatus.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketPolicyStatus.java @@ -41,10 +41,7 @@ public void process() throws GWException { logger.info(GWConstants.LOG_GET_BUCKET_POLICY_STATUS_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketReplication.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketReplication.java index 94198645..534f1878 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketReplication.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketReplication.java @@ -15,6 +15,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketReplication; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -37,17 +38,19 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + DataGetBucketReplication dataGetBucketReplication = new DataGetBucketReplication(s3Parameter); + dataGetBucketReplication.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_GET_REPLICATION_CONFIGURATION, s3Parameter, dataGetBucketReplication)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } String replication = getBucketInfo().getReplication(); logger.debug(GWConstants.LOG_GET_BUCKET_REPLICATION, replication); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketTagIndex.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketTagIndex.java new file mode 100644 index 00000000..e299c315 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketTagIndex.java @@ -0,0 +1,77 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import java.io.IOException; + +import jakarta.servlet.http.HttpServletResponse; + +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketTagging; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; + +import org.slf4j.LoggerFactory; + +public class GetBucketTagIndex extends S3Request { + public GetBucketTagIndex(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(GetBucketTagIndex.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_GET_BUCKET_TAG_INDEX_START); + + String bucket = s3Parameter.getBucketName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + // DataGetBucketTagging dataGetBucketTagging = new DataGetBucketTagging(s3Parameter); + // dataGetBucketTagging.extract(); + + // if (!checkPolicyBucket(GWConstants.ACTION_GET_BUCKET_TAGGING, s3Parameter, dataGetBucketTagging)) { + // checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + // } + + boolean enable = isBucketTagIndex(bucket); + + String xml = null; + + if (enable) { + xml = GWConstants.TAG_INDEX_CONFIGURATION_XMLNS_ENABLED; + } else { + xml = GWConstants.TAG_INDEX_CONFIGURATION_XMLNS_DISABLE; + } + + logger.debug(GWConstants.LOG_GET_BUCKET_TAG_INDEX_XML, xml); + try { + s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); + s3Parameter.getResponse().getOutputStream().write(xml.getBytes()); + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketTagging.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketTagging.java index 621962fb..797fc984 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketTagging.java @@ -13,6 +13,7 @@ import java.io.IOException; import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketTagging; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -35,17 +36,19 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + DataGetBucketTagging dataGetBucketTagging = new DataGetBucketTagging(s3Parameter); + dataGetBucketTagging.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_GET_BUCKET_TAGGING, s3Parameter, dataGetBucketTagging)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } String tag = getBucketInfo().getTagging(); logger.debug(GWConstants.LOG_TAGGING, tag); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketVersioning.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketVersioning.java index f171d5dd..d33395bb 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketVersioning.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketVersioning.java @@ -15,6 +15,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketVersioning; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -37,17 +38,19 @@ public void process() throws GWException { logger.info(GWConstants.LOG_GET_BUCKET_VERSIONING_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + DataGetBucketVersioning dataGetBucketVersioning = new DataGetBucketVersioning(s3Parameter); + dataGetBucketVersioning.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_GET_BUCKET_VERSIONING, s3Parameter, dataGetBucketVersioning)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } String versioningStatus = getBucketInfo().getVersioning(); logger.debug(GWConstants.LOG_GET_BUCKET_VERSIONING, bucket, versioningStatus); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketWebsite.java b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketWebsite.java index 15a5a060..8c05880f 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetBucketWebsite.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetBucketWebsite.java @@ -15,6 +15,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetBucketWebsite; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -37,16 +38,16 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } + DataGetBucketWebsite dataGetBucketWebsite = new DataGetBucketWebsite(s3Parameter); + dataGetBucketWebsite.extract(); + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); String web = getBucketInfo().getWeb(); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetObject.java b/core/src/com/pspace/ifs/ksan/gw/api/GetObject.java index 218bfed1..e8bfd02f 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetObject.java @@ -35,6 +35,7 @@ import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.gw.utils.GWConfig; import org.slf4j.LoggerFactory; @@ -54,10 +55,6 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); logger.debug(GWConstants.LOG_BUCKET_OBJECT, bucket, object); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -66,25 +63,46 @@ public void process() throws GWException { DataGetObject dataGetObject = new DataGetObject(s3Parameter); dataGetObject.extract(); - + String versionId = dataGetObject.getVersionId(); + // String versionId = null; + // String range = null; String range = dataGetObject.getRange(); String ifMatch = dataGetObject.getIfMatch(); String ifNoneMatch = dataGetObject.getIfNoneMatch(); String ifModifiedSince = dataGetObject.getIfModifiedSince(); String ifUnmodifiedSince = dataGetObject.getIfUnmodifiedSince(); + // long dbStart = System.currentTimeMillis(); Metadata objMeta = null; if (Strings.isNullOrEmpty(versionId)) { objMeta = open(bucket, object); - versionId = objMeta.getVersionId(); } else { objMeta = open(bucket, object, versionId); } + if (GWConfig.getInstance().isDBOP()) { + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + return; + } + // long dbEnd = System.currentTimeMillis(); + // logger.error("get - db op : {}", dbEnd - dbStart); + logger.debug(GWConstants.LOG_OBJECT_META, objMeta.toString()); - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); - checkGrantObject(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + s3Parameter.setTaggingInfo(objMeta.getTag()); + + if (Strings.isNullOrEmpty(versionId)) { + if (!checkPolicyBucket(GWConstants.ACTION_GET_OBJECT, s3Parameter, dataGetObject)) { + checkGrantObject(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + } else { + if (!checkPolicyBucket(GWConstants.ACTION_GET_OBJECT_VERSION, s3Parameter, dataGetObject)) { + checkGrantObject(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + } + + versionId = objMeta.getVersionId(); + s3Parameter.setVersionId(versionId); S3Metadata s3Metadata = null; @@ -158,6 +176,13 @@ public void process() throws GWException { throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } + if (GWConfig.getInstance().isS3OP()) { + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + return; + } + + // long fileEnd = System.currentTimeMillis(); + // logger.error("get op : {}", fileEnd - dbEnd); s3Parameter.getResponse().setStatus(resultRange.getStatus()); } @@ -194,7 +219,6 @@ public void addMetadataToResponse(HttpServletResponse response, S3Metadata metad HttpHeaders.CONTENT_TYPE, GWConstants.RESPONSE_CONTENT_TYPE, metadata.getContentType()); - // TODO: handles only a single range due to jclouds limitations Collection contentRanges = contentsHeaders; if (contentsHeaders != null && !contentRanges.isEmpty()) { for (String contents : contentsHeaders) { @@ -227,15 +251,15 @@ public void addMetadataToResponse(HttpServletResponse response, S3Metadata metad response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION, metadata.getServersideEncryption()); } - if (metadata.getLockMode() != null) { + if (!Strings.isNullOrEmpty(metadata.getLockMode())) { response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_MODE, metadata.getLockMode()); } - if (metadata.getLockExpires() != null) { + if (!Strings.isNullOrEmpty(metadata.getLockExpires())) { response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, metadata.getLockExpires()); } - if (metadata.getLegalHold() != null) { + if (!Strings.isNullOrEmpty(metadata.getLegalHold())) { response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_LEGAL_HOLD, metadata.getLegalHold()); } diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetObjectAcl.java b/core/src/com/pspace/ifs/ksan/gw/api/GetObjectAcl.java index 96c3fb3f..c3a880f6 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetObjectAcl.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetObjectAcl.java @@ -43,10 +43,6 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); logger.debug(GWConstants.LOG_BUCKET_OBJECT, bucket, object); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -56,6 +52,7 @@ public void process() throws GWException { DataGetObjectAcl dataGetObjectAcl = new DataGetObjectAcl(s3Parameter); dataGetObjectAcl.extract(); String versionId = dataGetObjectAcl.getVersionId(); + s3Parameter.setVersionId(versionId); Metadata objMeta = null; if (Strings.isNullOrEmpty(versionId)) { @@ -64,9 +61,17 @@ public void process() throws GWException { objMeta = open(bucket, object, versionId); } logger.debug(GWConstants.LOG_OBJECT_META, objMeta.toString()); - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); + s3Parameter.setTaggingInfo(objMeta.getTag()); - checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + if (Strings.isNullOrEmpty(versionId)) { + if (!checkPolicyBucket(GWConstants.ACTION_GET_OBJECT_ACL, s3Parameter, dataGetObjectAcl)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } + } else { + if (!checkPolicyBucket(GWConstants.ACTION_GET_OBJECT_VERSION_ACL, s3Parameter, dataGetObjectAcl)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); + } + } String aclInfo = objMeta.getAcl(); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetObjectLegalHold.java b/core/src/com/pspace/ifs/ksan/gw/api/GetObjectLegalHold.java new file mode 100644 index 00000000..88724e2b --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetObjectLegalHold.java @@ -0,0 +1,124 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import java.io.IOException; + +import jakarta.servlet.http.HttpServletResponse; +import javax.xml.stream.XMLOutputFactory; +import javax.xml.stream.XMLStreamException; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.gw.data.DataGetObjectLegalHold; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.format.ObjectLockConfiguration; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; + +import org.slf4j.LoggerFactory; + +public class GetObjectLegalHold extends S3Request { + + public GetObjectLegalHold(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(GetObjectLegalHold.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_GET_OBJECT_LOCK_CONFIGURATION_START); + + String bucket = s3Parameter.getBucketName(); + initBucketInfo(bucket); + String object = s3Parameter.getObjectName(); + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + DataGetObjectLegalHold dataGetObjectLegalHold = new DataGetObjectLegalHold(s3Parameter); + dataGetObjectLegalHold.extract(); + + String versionId = dataGetObjectLegalHold.getVersionId(); + s3Parameter.setVersionId(versionId); + + Metadata objMeta = null; + if (Strings.isNullOrEmpty(versionId)) { + objMeta = open(bucket, object); + } else { + objMeta = open(bucket, object, versionId); + } + + logger.debug(GWConstants.LOG_OBJECT_META, objMeta.toString()); + s3Parameter.setTaggingInfo(objMeta.getTag()); + if (!checkPolicyBucket(GWConstants.ACTION_GET_OBJECT_LEGAL_HOLD, s3Parameter, dataGetObjectLegalHold)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + + String objectLock = getBucketInfo().getObjectLock(); + logger.debug(GWConstants.LOG_OBJECT_LOCK, objectLock); + if (Strings.isNullOrEmpty(objectLock)) { + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); + } + + try { + ObjectLockConfiguration oc = new XmlMapper().readValue(objectLock, ObjectLockConfiguration.class); + if (!GWConstants.STATUS_ENABLED.equals(oc.objectLockEnabled)) { + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); + } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + String meta = objMeta.getMeta(); + S3Metadata s3Metadata; + try { + s3Metadata = new ObjectMapper().readValue(meta, S3Metadata.class); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + String legalHold = s3Metadata.getLegalHold(); + if (!Strings.isNullOrEmpty(legalHold)) { + XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newFactory(); + javax.xml.stream.XMLStreamWriter xml; + try { + xml = xmlOutputFactory.createXMLStreamWriter(s3Parameter.getResponse().getOutputStream()); + xml.writeStartDocument(); + xml.writeStartElement(GWConstants.LEGAL_HOLD); + writeSimpleElement(xml, GWConstants.XML_STATUS, legalHold); + xml.writeEndElement(); + xml.flush(); + } catch (XMLStreamException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } else { + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); + } + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } + +} diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetObjectRetention.java b/core/src/com/pspace/ifs/ksan/gw/api/GetObjectRetention.java index a88f1d1b..10586bbf 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetObjectRetention.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetObjectRetention.java @@ -18,8 +18,10 @@ import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; import com.google.common.base.Strings; import com.pspace.ifs.ksan.gw.data.DataGetObjectRetention; +import com.pspace.ifs.ksan.gw.format.ObjectLockConfiguration; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Bucket; @@ -48,10 +50,6 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); logger.debug(GWConstants.LOG_BUCKET_OBJECT, bucket, object); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -61,6 +59,7 @@ public void process() throws GWException { DataGetObjectRetention dataGetObjectRetention = new DataGetObjectRetention(s3Parameter); dataGetObjectRetention.extract(); String versionId = dataGetObjectRetention.getVersionId(); + s3Parameter.setVersionId(versionId); Metadata objMeta = null; if (Strings.isNullOrEmpty(versionId)) { @@ -70,9 +69,27 @@ public void process() throws GWException { } logger.debug(GWConstants.LOG_OBJECT_META, objMeta.toString()); - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); + s3Parameter.setTaggingInfo(objMeta.getTag()); - checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + if (!checkPolicyBucket(GWConstants.ACTION_GET_OBJECT_RETENTION, s3Parameter, dataGetObjectRetention)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + + try { + String objectLock = getBucketInfo().getObjectLock(); + if (Strings.isNullOrEmpty(objectLock)) { + logger.info("bucket objectlock is null or empty."); + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); + } + ObjectLockConfiguration oc = new XmlMapper().readValue(objectLock, ObjectLockConfiguration.class); + if (!oc.objectLockEnabled.equals(GWConstants.STATUS_ENABLED)) { + logger.info("bucket objectlock is not equals Enabled."); + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); + } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } String meta = objMeta.getMeta(); S3Metadata s3Metadata; diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetObjectTagging.java b/core/src/com/pspace/ifs/ksan/gw/api/GetObjectTagging.java index a2d2bc19..25e9fe17 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetObjectTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetObjectTagging.java @@ -40,10 +40,7 @@ public void process() throws GWException { logger.info(GWConstants.LOG_GET_OBJECT_TAGGING_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -56,6 +53,7 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); String versionId = dataGetObjectTagging.getVersionId(); + s3Parameter.setVersionId(versionId); Metadata objMeta = null; if (Strings.isNullOrEmpty(versionId)) { @@ -64,10 +62,17 @@ public void process() throws GWException { objMeta = open(bucket, object, versionId); } - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); + s3Parameter.setTaggingInfo(objMeta.getTag()); + if (Strings.isNullOrEmpty(versionId)) { + if (!checkPolicyBucket(GWConstants.ACTION_GET_OBJECT_TAGGING, s3Parameter, dataGetObjectTagging)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + } else { + if (!checkPolicyBucket(GWConstants.ACTION_GET_OBJECT_VERSION_TAGGING, s3Parameter, dataGetObjectTagging)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + } - checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); - String taggingInfo = objMeta.getTag(); logger.info(GWConstants.LOG_TAGGING, taggingInfo); if ( Strings.isNullOrEmpty(taggingInfo)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/GetPublicAccessBlock.java b/core/src/com/pspace/ifs/ksan/gw/api/GetPublicAccessBlock.java index 73c16009..d840b2a8 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/GetPublicAccessBlock.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/GetPublicAccessBlock.java @@ -37,10 +37,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/HeadBucket.java b/core/src/com/pspace/ifs/ksan/gw/api/HeadBucket.java index a12d2631..ad5c2a8c 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/HeadBucket.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/HeadBucket.java @@ -36,10 +36,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/HeadObject.java b/core/src/com/pspace/ifs/ksan/gw/api/HeadObject.java index 465ad06c..e18ed4bb 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/HeadObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/HeadObject.java @@ -10,11 +10,18 @@ */ package com.pspace.ifs.ksan.gw.api; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Date; + +import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletResponse; import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.ObjectMapper; import com.google.common.base.Strings; +import com.google.common.net.HttpHeaders; import com.pspace.ifs.ksan.gw.data.DataHeadObject; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; @@ -25,10 +32,11 @@ import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.gw.utils.GWConfig; import org.slf4j.LoggerFactory; -public class HeadObject extends S3Request { +public class HeadObject extends S3Request implements S3AddResponse { public HeadObject(S3Parameter s3Parameter) { super(s3Parameter); @@ -41,10 +49,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -55,6 +60,7 @@ public void process() throws GWException { dataHeadObject.extract(); String versionId = dataHeadObject.getVersionId(); + s3Parameter.setVersionId(versionId); String expectedBucketOwner = dataHeadObject.getExpectedBucketOwner(); if (!Strings.isNullOrEmpty(expectedBucketOwner)) { @@ -71,7 +77,12 @@ public void process() throws GWException { objMeta = open(bucket, object, versionId); } - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); + if (GWConfig.getInstance().isDBOP()) { + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + return; + } + + // objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); checkGrantObject(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); // meta info @@ -85,7 +96,7 @@ public void process() throws GWException { if (!Strings.isNullOrEmpty(dataHeadObject.getServerSideEncryptionCustomerKey())) { if (!s3Metadata.getCustomerKey().equals(dataHeadObject.getServerSideEncryptionCustomerKey())) { logger.warn(GWConstants.ENCRYPTION_CUSTOMER_KEY_IS_INVALID); - throw new GWException(GWErrorCode.BAD_REQUEST, s3Parameter); + throw new GWException(GWErrorCode.BAD_REQUEST, s3Parameter); } } else { logger.warn(GWConstants.ENCRYPTION_CUSTOMER_KEY_IS_NULL); @@ -94,7 +105,7 @@ public void process() throws GWException { } s3Parameter.getResponse().addHeader(GWConstants.X_AMZ_VERSION_ID, s3Metadata.getVersionId()); - GWUtils.addMetadataToResponse(s3Parameter.getRequest(), s3Parameter.getResponse(), s3Metadata, null, null); + addMetadataToResponse(s3Parameter.getResponse(), s3Metadata, null, null); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); @@ -102,4 +113,118 @@ public void process() throws GWException { s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); } + + @Override + public void addResponseHeaderWithOverride(HttpServletRequest request, HttpServletResponse response, + String headerName, String overrideHeaderName, String value) { + + String override = request.getParameter(overrideHeaderName); + + // NPE in if value is null + override = (!Strings.isNullOrEmpty(override)) ? override : value; + + if (!Strings.isNullOrEmpty(override)) { + response.addHeader(headerName, override); + } + } + + @Override + public void addMetadataToResponse(HttpServletResponse response, S3Metadata metadata, List contentsHeaders, + Long streamSize) { + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CACHE_CONTROL, GWConstants.RESPONSE_CACHE_CONTROL, + metadata.getCacheControl()); + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CONTENT_ENCODING, GWConstants.RESPONSE_CONTENT_ENCODING, + metadata.getContentEncoding()); + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CONTENT_LANGUAGE, GWConstants.RESPONSE_CONTENT_LANGUAGE, + metadata.getContentLanguage()); + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CONTENT_DISPOSITION, GWConstants.RESPONSE_CONTENT_DISPOSITION, + metadata.getContentDisposition()); + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CONTENT_TYPE, GWConstants.RESPONSE_CONTENT_TYPE, + metadata.getContentType()); + + Collection contentRanges = contentsHeaders; + if (contentsHeaders != null && !contentRanges.isEmpty()) { + for (String contents : contentsHeaders) { + response.addHeader(HttpHeaders.CONTENT_RANGE, contents); + } + + response.addHeader(HttpHeaders.ACCEPT_RANGES, GWConstants.BYTES); + response.addHeader(HttpHeaders.CONTENT_LENGTH, streamSize.toString()); + } else { + response.addHeader(HttpHeaders.CONTENT_LENGTH, metadata.getContentLength().toString()); + logger.debug(GWConstants.LOG_GET_OBJECT_CONTENT_LENGTH, metadata.getContentLength()); + } + + String overrideContentType = s3Parameter.getRequest().getParameter(GWConstants.RESPONSE_CONTENT_TYPE); + response.setContentType(overrideContentType != null ? overrideContentType : metadata.getContentType()); + + if (!Strings.isNullOrEmpty(metadata.getCustomerAlgorithm())) { + response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, metadata.getCustomerAlgorithm()); + } + + if (!Strings.isNullOrEmpty(metadata.getCustomerKey())) { + response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, metadata.getCustomerKey()); + } + + if (!Strings.isNullOrEmpty(metadata.getCustomerKeyMD5())) { + response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, metadata.getCustomerKeyMD5()); + } + + if (!Strings.isNullOrEmpty(metadata.getServersideEncryption())) { + response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION, metadata.getServersideEncryption()); + } + + if (!Strings.isNullOrEmpty(metadata.getLockMode())) { + response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_MODE, metadata.getLockMode()); + } + + if (!Strings.isNullOrEmpty(metadata.getLockExpires())) { + response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, metadata.getLockExpires()); + } + + if (!Strings.isNullOrEmpty(metadata.getLegalHold())) { + response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_LEGAL_HOLD, metadata.getLegalHold()); + } + + if (metadata.getUserMetadataMap() != null) { + for (Map.Entry entry : metadata.getUserMetadataMap().entrySet()) { + response.addHeader(entry.getKey(), entry.getValue()); + logger.debug(GWConstants.LOG_GET_OBJECT_USER_META_DATA, entry.getKey(), entry.getValue()); + } + } + + response.addHeader(HttpHeaders.ETAG, GWUtils.maybeQuoteETag(metadata.getETag())); + + String overrideExpires = s3Parameter.getRequest().getParameter(GWConstants.RESPONSE_EXPIRES); + if (overrideExpires != null) { + response.addHeader(HttpHeaders.EXPIRES, overrideExpires); + } else { + Date expires = metadata.getExpires(); + if (expires != null) { + response.addDateHeader(HttpHeaders.EXPIRES, expires.getTime()); + } + } + + logger.debug(GWConstants.LOG_GET_OBJECT_MODIFIED, metadata.getLastModified().getTime()); + response.addDateHeader(HttpHeaders.LAST_MODIFIED, metadata.getLastModified().getTime()); + + if (!Strings.isNullOrEmpty(metadata.getTaggingCount())) { + response.addHeader(GWConstants.X_AMZ_TAGGING_COUNT, metadata.getTaggingCount()); + } + + if (!Strings.isNullOrEmpty(metadata.getVersionId())) { + if (metadata.getVersionId().equalsIgnoreCase(GWConstants.VERSIONING_DISABLE_TAIL)) { + response.addHeader(GWConstants.X_AMZ_VERSION_ID, GWConstants.VERSIONING_DISABLE_TAIL); + } else { + response.addHeader(GWConstants.X_AMZ_VERSION_ID, metadata.getVersionId()); + } + } else { + response.addHeader(GWConstants.X_AMZ_VERSION_ID, GWConstants.VERSIONING_DISABLE_TAIL); + } + } } \ No newline at end of file diff --git a/core/src/com/pspace/ifs/ksan/gw/api/ListBucketTagSearch.java b/core/src/com/pspace/ifs/ksan/gw/api/ListBucketTagSearch.java new file mode 100644 index 00000000..276f7f41 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/ListBucketTagSearch.java @@ -0,0 +1,127 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import java.io.IOException; +import java.io.Writer; +import java.util.Map.Entry; +import java.util.List; + +import javax.xml.stream.XMLOutputFactory; +import javax.xml.stream.XMLStreamException; +import javax.xml.stream.XMLStreamWriter; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataListBucketTagSearch; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.libs.identity.ObjectListParameter; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.libs.identity.S3ObjectList; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; + +import org.slf4j.LoggerFactory; + +public class ListBucketTagSearch extends S3Request { + + public ListBucketTagSearch(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(ListBucketTagSearch.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_LIST_BUCKET_TAG_SEARCH_START); + + String bucket = s3Parameter.getBucketName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + DataListBucketTagSearch dataListBucketTagSearch = new DataListBucketTagSearch(s3Parameter); + dataListBucketTagSearch.extract(); + + logger.info("bucket policy : {}", s3Parameter.getBucket().getPolicy()); + if (!checkPolicyBucket(GWConstants.ACTION_LIST_BUCKET, s3Parameter, dataListBucketTagSearch)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + + String encodingType = dataListBucketTagSearch.getEncodingType(); + String maxKeys = dataListBucketTagSearch.getMaxkeys(); + int max = 1000; + + if (!Strings.isNullOrEmpty(maxKeys)) { + max = Integer.parseInt(maxKeys); + } + + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); + XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); + + List tagList = listBucketTags(bucket, dataListBucketTagSearch.getTag(), max); + + logger.info("tag : {}", dataListBucketTagSearch.getTag()); + logger.info("tagList count : {}", tagList.size()); + try (Writer writer = s3Parameter.getResponse().getWriter()) { + s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); + XMLStreamWriter xmlStreamWriter = xmlOutputFactory.createXMLStreamWriter(writer); + xmlStreamWriter.writeStartDocument(); + xmlStreamWriter.writeStartElement(GWConstants.LIST_BUCKET_TAG_SEARCH); + xmlStreamWriter.writeDefaultNamespace(GWConstants.AWS_XMLNS); + + for (Metadata meta : tagList) { + S3Metadata s3Metadata = new S3Metadata(); + ObjectMapper jsonMapper = new ObjectMapper(); + + try { + s3Metadata = jsonMapper.readValue(meta.getMeta(), S3Metadata.class); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + xmlStreamWriter.writeStartElement(GWConstants.XML_CONTENTS); + logger.debug("meta info : {}, {}, {}, {}, {}, {}, {}, {}", meta.getPath(), formatDate(s3Metadata.getLastModified()), s3Metadata.getETag(), s3Metadata.getContentLength(), s3Metadata.getTier(), meta.getTag(), s3Metadata.getOwnerId(), s3Metadata.getOwnerName()); + + writeSimpleElement(xmlStreamWriter, GWConstants.KEY, GWUtils.encodeObjectName(encodingType, meta.getPath())); + if (s3Metadata.getLastModified() != null) { + writeSimpleElement(xmlStreamWriter, GWConstants.LAST_MODIFIED, formatDate(s3Metadata.getLastModified())); + } + + if (s3Metadata.getETag() != null) { + writeSimpleElement(xmlStreamWriter, GWConstants.ETAG, GWUtils.maybeQuoteETag(s3Metadata.getETag())); + } + + writeSimpleElement(xmlStreamWriter, GWConstants.XML_SIZE, s3Metadata.getContentLength().toString()); + writeSimpleElement(xmlStreamWriter, GWConstants.STORAGE_CLASS, s3Metadata.getTier()); + // writeSimpleElement(xmlStreamWriter, GWConstants.XML_TAG, meta.getTag()); + writeOwnerInfini(xmlStreamWriter, s3Metadata.getOwnerId(), s3Metadata.getOwnerName()); + xmlStreamWriter.writeEndElement(); + } + + xmlStreamWriter.writeEndElement(); + xmlStreamWriter.flush(); + } catch (IOException | XMLStreamException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/api/ListMultipartUploads.java b/core/src/com/pspace/ifs/ksan/gw/api/ListMultipartUploads.java index 1ca34df3..545196fa 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/ListMultipartUploads.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/ListMultipartUploads.java @@ -27,6 +27,7 @@ import com.pspace.ifs.ksan.libs.multipart.ResultUploads; import com.pspace.ifs.ksan.libs.multipart.Upload; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.objmanager.ObjMultipart; @@ -46,21 +47,20 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); DataListMultipartUploads dataListMultipartUploads = new DataListMultipartUploads(s3Parameter); dataListMultipartUploads.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_LIST_BUCKET_MULTIPART_UPLOADS, s3Parameter, dataListMultipartUploads)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + String delimiter = dataListMultipartUploads.getDelimiter(); String encodingType = dataListMultipartUploads.getEncodingType(); String prefix = dataListMultipartUploads.getPrefix(); @@ -86,7 +86,7 @@ public void process() throws GWException { throw new GWException(GWErrorCode.INTERNAL_SERVER_ERROR, s3Parameter); } - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); try (Writer writer = s3Parameter.getResponse().getWriter()) { s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/ListObjectVersions.java b/core/src/com/pspace/ifs/ksan/gw/api/ListObjectVersions.java index 7230801c..142699c4 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/ListObjectVersions.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/ListObjectVersions.java @@ -28,6 +28,7 @@ import com.pspace.ifs.ksan.libs.identity.S3ObjectList; import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; @@ -46,20 +47,20 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); - DataListObjectVersions dataListObjectVersions = new DataListObjectVersions(s3Parameter); dataListObjectVersions.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_LIST_BUCKET_VERSIONS, s3Parameter, dataListObjectVersions)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + String delimiter = dataListObjectVersions.getDelimiter(); String encodingType = dataListObjectVersions.getEncodingType(); String keyMarker = dataListObjectVersions.getKeyMarker(); @@ -95,7 +96,7 @@ public void process() throws GWException { s3ObjectList.setPrefix(prefix); s3ObjectList.setVersionIdMarker(versionIdMarker); - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); ObjectListParameter objectListParameter = listObjectVersions(bucket, s3ObjectList); @@ -172,7 +173,11 @@ public void process() throws GWException { writeOwnerInfini(xmlStreamWriter, s3Metadata.getOwnerId(), s3Metadata.getOwnerName()); xmlStreamWriter.writeEndElement(); - logger.debug(GWConstants.LOG_LIST_OBJECT_VERSIONS_INFO, s3Metadata.getName(), s3Metadata.getLastModified(), s3Metadata.getVersionId()); + if (s3Metadata.getDeleteMarker().compareTo(GWConstants.OBJECT_TYPE_MARK) == 0) { + logger.debug(GWConstants.LOG_LIST_OBJECT_VERSIONS_MARKER, s3Metadata.getName(), s3Metadata.getLastModified(), s3Metadata.getVersionId()); + } else { + logger.debug(GWConstants.LOG_LIST_OBJECT_VERSIONS_INFO, s3Metadata.getName(), s3Metadata.getLastModified(), s3Metadata.getVersionId()); + } } } } diff --git a/core/src/com/pspace/ifs/ksan/gw/api/ListObjects.java b/core/src/com/pspace/ifs/ksan/gw/api/ListObjects.java index f684920b..e2e9b4a5 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/ListObjects.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/ListObjects.java @@ -28,6 +28,7 @@ import com.pspace.ifs.ksan.libs.identity.S3ObjectList; import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; @@ -46,21 +47,21 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); - DataListBuckets dataListBuckets = new DataListBuckets(s3Parameter); dataListBuckets.extract(); + logger.info("bucket policy : {}", s3Parameter.getBucket().getPolicy()); + if (!checkPolicyBucket(GWConstants.ACTION_LIST_BUCKET, s3Parameter, dataListBuckets)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + S3ObjectList s3ObjectList = new S3ObjectList(); if (!Strings.isNullOrEmpty(dataListBuckets.getMaxkeys())) { try { @@ -86,7 +87,7 @@ public void process() throws GWException { logger.debug("marker : {}", dataListBuckets.getMarker()); logger.debug("prefix : {}", dataListBuckets.getPrefix()); logger.debug("maxKeys : {}", dataListBuckets.getMaxkeys()); - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); ObjectListParameter objectListParameter = listObject(bucket, s3ObjectList); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/ListObjectsV2.java b/core/src/com/pspace/ifs/ksan/gw/api/ListObjectsV2.java index 09ce6326..48d6630f 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/ListObjectsV2.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/ListObjectsV2.java @@ -28,6 +28,7 @@ import com.pspace.ifs.ksan.libs.identity.S3ObjectList; import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; @@ -46,21 +47,20 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); DataListObjectV2 dataListObjectV2 = new DataListObjectV2(s3Parameter); dataListObjectV2.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_LIST_BUCKET, s3Parameter, dataListObjectV2)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + // read header S3ObjectList s3ObjectList = new S3ObjectList(); @@ -87,7 +87,7 @@ public void process() throws GWException { s3ObjectList.setStartAfter(dataListObjectV2.getStartAfter()); s3ObjectList.setFetchOwner(dataListObjectV2.getFetchOwner()); - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/ListParts.java b/core/src/com/pspace/ifs/ksan/gw/api/ListParts.java index 2fe0584f..c77c1103 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/ListParts.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/ListParts.java @@ -34,6 +34,7 @@ import com.pspace.ifs.ksan.libs.multipart.Part; import com.pspace.ifs.ksan.libs.multipart.ResultParts; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.objmanager.ObjMultipart; @@ -57,20 +58,20 @@ public void process() throws GWException { throw new GWException(GWErrorCode.NO_SUCH_BUCKET, s3Parameter); } initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); - + DataListParts dataListParts = new DataListParts(s3Parameter); dataListParts.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_LIST_MULTIPART_UPLOAD_PARTS, s3Parameter, dataListParts)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); + } + String maxParts = dataListParts.getMaxParts(); String partNumberMarker = dataListParts.getPartNumberMarker(); String uploadId = dataListParts.getUploadId(); @@ -120,7 +121,7 @@ public void process() throws GWException { XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); try (Writer writer = s3Parameter.getResponse().getWriter()) { s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); XMLStreamWriter xmlStreamWriter = xmlOutputFactory.createXMLStreamWriter(writer); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/OptionsObject.java b/core/src/com/pspace/ifs/ksan/gw/api/OptionsObject.java index 59644c2c..f4eab9c2 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/OptionsObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/OptionsObject.java @@ -24,6 +24,7 @@ import com.pspace.ifs.ksan.libs.PrintStack; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; import org.slf4j.LoggerFactory; diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PostObject.java b/core/src/com/pspace/ifs/ksan/gw/api/PostObject.java index 360e63c9..191d16d8 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PostObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PostObject.java @@ -19,6 +19,7 @@ import java.util.Base64.Decoder; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.dataformat.xml.XmlMapper; import com.google.common.base.Strings; @@ -61,9 +62,6 @@ public void process() throws GWException { initBucketInfo(bucket); S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); DataPostObject dataPostObject = new DataPostObject(s3Parameter); dataPostObject.extract(); @@ -271,7 +269,8 @@ public void process() throws GWException { dataPostObject.getGrantFullControl(), dataPostObject.getGrantReadAcp(), dataPostObject.getGrantWriteAcp(), - s3Parameter); + s3Parameter, + false); String bucketEncryption = getBucketInfo().getEncryption(); S3ServerSideEncryption encryption = new S3ServerSideEncryption(bucketEncryption, s3Metadata, s3Parameter); @@ -332,6 +331,7 @@ public void process() throws GWException { } else { versionId = GWConstants.VERSIONING_DISABLE_TAIL; } + s3Parameter.setVersionId(versionId); } catch (GWException e) { logger.info(e.getMessage()); if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { @@ -362,6 +362,7 @@ public void process() throws GWException { ObjectMapper jsonMapper = new ObjectMapper(); String jsonmeta = ""; try { + // jsonMapper.setSerializationInclusion(Include.NON_NULL); jsonmeta = jsonMapper.writeValueAsString(s3Metadata); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketAcl.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketAcl.java index 0784a34c..a69e7b8e 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketAcl.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketAcl.java @@ -13,7 +13,7 @@ import java.util.ArrayList; import jakarta.servlet.http.HttpServletResponse; - +import com.google.common.base.Strings; import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.JsonMappingException; import com.fasterxml.jackson.dataformat.xml.XmlMapper; @@ -43,21 +43,20 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_ACL_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); DataPutBucketAcl dataPutBucketAcl = new DataPutBucketAcl(s3Parameter); dataPutBucketAcl.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_BUCKET_ACL, s3Parameter, dataPutBucketAcl)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + AccessControlPolicy preAccessControlPolicy = null; try { @@ -75,8 +74,6 @@ public void process() throws GWException { accessControlPolicy.aclList = new AccessControlList(); accessControlPolicy.aclList.grants = new ArrayList(); accessControlPolicy.owner = new Owner(); - accessControlPolicy.owner.id = s3Parameter.getUser().getUserId(); - accessControlPolicy.owner.displayName = s3Parameter.getUser().getUserName(); String xml = GWUtils.makeAclXml(accessControlPolicy, preAccessControlPolicy, @@ -91,7 +88,8 @@ public void process() throws GWException { dataPutBucketAcl.getGrantFullControl(), dataPutBucketAcl.getGrantReadAcp(), dataPutBucketAcl.getGrantWriteAcp(), - s3Parameter); + s3Parameter, + true); logger.debug(GWConstants.LOG_ACL, xml); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketCors.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketCors.java index aa0e9e91..cb2c1da0 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketCors.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketCors.java @@ -33,21 +33,20 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_CORS_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); - + DataPutBucketCors bucketCors = new DataPutBucketCors(s3Parameter); bucketCors.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_BUCKET_CORS, s3Parameter, bucketCors)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + String corsInfo = bucketCors.getCorsXml(); updateBucketCors(bucket, corsInfo); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketEncryption.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketEncryption.java index b5597c4c..92097be6 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketEncryption.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketEncryption.java @@ -34,21 +34,20 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_ENCRYPTION_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); - + DataPutBucketEncryption dataPutBucketEncryption = new DataPutBucketEncryption(s3Parameter); dataPutBucketEncryption.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_ENCRYPTION_CONFIGURATION, s3Parameter, dataPutBucketEncryption)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + String encryptionInfo = dataPutBucketEncryption.getEncryptionXml(); updateBucketEncryption(bucket, encryptionInfo); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketLifecycleConfiguration.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketLifecycleConfiguration.java index 36c87dd7..746f1660 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketLifecycleConfiguration.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketLifecycleConfiguration.java @@ -33,21 +33,20 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_LIFECYCLE_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); DataPutBucketLifeCycle dataPutBucketLifeCycle = new DataPutBucketLifeCycle(s3Parameter); dataPutBucketLifeCycle.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_LIFECYCLE_CONFIGURATION, s3Parameter, dataPutBucketLifeCycle)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + String lifecycleXml = dataPutBucketLifeCycle.getLifecycleXml(); logger.info(GWConstants.LOG_PUT_BUCKET_LIFECYCLE_XML, lifecycleXml); updateBucketLifecycle(bucket, lifecycleXml); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketLogging.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketLogging.java new file mode 100644 index 00000000..1434d086 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketLogging.java @@ -0,0 +1,60 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import java.io.IOException; + +import jakarta.servlet.http.HttpServletResponse; + +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.pspace.ifs.ksan.gw.data.DataPutBucketLogging; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; + +import org.slf4j.LoggerFactory; + +public class PutBucketLogging extends S3Request { + + public PutBucketLogging(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(PutBucketLogging.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_PUT_BUCKET_LOGGING_START); + String bucket = s3Parameter.getBucketName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + DataPutBucketLogging dataPutBucketLogging = new DataPutBucketLogging(s3Parameter); + dataPutBucketLogging.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_PUT_BUCKET_LOGGING, s3Parameter, dataPutBucketLogging)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + + String loggingInfo = dataPutBucketLogging.getLoggingXml(); + updateBucketLogging(bucket, loggingInfo); + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutObjectLockConfiguration.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketObjectLock.java similarity index 89% rename from core/src/com/pspace/ifs/ksan/gw/api/PutObjectLockConfiguration.java rename to core/src/com/pspace/ifs/ksan/gw/api/PutBucketObjectLock.java index f3a3a36b..5fbc77e5 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutObjectLockConfiguration.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketObjectLock.java @@ -15,46 +15,45 @@ import jakarta.servlet.http.HttpServletResponse; import com.fasterxml.jackson.dataformat.xml.XmlMapper; -import com.google.common.base.Strings; import com.pspace.ifs.ksan.gw.data.DataPutBucketObjectLock; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; -import com.pspace.ifs.ksan.gw.format.ObjectLockConfiguration; import com.pspace.ifs.ksan.gw.identity.S3Bucket; import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.libs.PrintStack; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.gw.format.ObjectLockConfiguration; +import com.google.common.base.Strings; import org.slf4j.LoggerFactory; -public class PutObjectLockConfiguration extends S3Request { +public class PutBucketObjectLock extends S3Request { - public PutObjectLockConfiguration(S3Parameter s3Parameter) { + public PutBucketObjectLock(S3Parameter s3Parameter) { super(s3Parameter); - logger = LoggerFactory.getLogger(PutObjectLockConfiguration.class); + logger = LoggerFactory.getLogger(PutBucketObjectLock.class); } - + @Override public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_OBJECT_LOCK_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); - + DataPutBucketObjectLock dataPutBucketObjectLock = new DataPutBucketObjectLock(s3Parameter); dataPutBucketObjectLock.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_BUCKET_OBJECT_LOCK_CONFIGURATION, s3Parameter, dataPutBucketObjectLock)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + String ObjectLockXml = dataPutBucketObjectLock.getObjectLockXml(); if(Strings.isNullOrEmpty(getBucketInfo().getObjectLock())) { @@ -115,5 +114,4 @@ public void process() throws GWException { s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); } - } diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketPolicy.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketPolicy.java index c800f443..f52a0f5a 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketPolicy.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketPolicy.java @@ -33,21 +33,20 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_POLICY_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); DataPutBucketPolicy dataPutBucketPolicy = new DataPutBucketPolicy(s3Parameter); dataPutBucketPolicy.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_BUCKET_POLICY, s3Parameter, dataPutBucketPolicy)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + String policyJson = dataPutBucketPolicy.getPolicyJson(); GWUtils.isPublicPolicyBucket(policyJson, s3Parameter); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketReplication.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketReplication.java index 67725316..4a67bc4e 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketReplication.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketReplication.java @@ -30,6 +30,7 @@ import com.pspace.ifs.ksan.gw.identity.S3Bucket; import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.objmanager.Bucket; @@ -47,25 +48,24 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_REPLICATION_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); - + if (!GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(getBucketVersioning(bucket))) { - throw new GWException(GWErrorCode.INVALID_REPLICATION_REQUEST, s3Parameter); + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); } DataPutBucketReplication dataPutBucketReplication = new DataPutBucketReplication(s3Parameter); dataPutBucketReplication.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_REPLICATION_CONFIGURATION, s3Parameter, dataPutBucketReplication)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + String replicationXml = dataPutBucketReplication.getReplicationXml(); checkBucketReplication(replicationXml); updateBucketReplication(bucket, replicationXml); @@ -103,7 +103,7 @@ private void checkBucketReplication(String replicationXml) throws GWException { } else { byte[] array = new byte[7]; // length is bounded by 7 new Random().nextBytes(array); - String generatedString = new String(array, Charset.forName(GWConstants.CHARSET_UTF_8)); + String generatedString = new String(array, Charset.forName(Constants.CHARSET_UTF_8)); id.put(generatedString, generatedString); } @@ -164,7 +164,7 @@ private void checkBucketReplication(String replicationXml) throws GWException { } if (Strings.isNullOrEmpty(arnPath[3])) { - if (isExistBucket(arnPath[5])) { + if (!isExistBucket(arnPath[5])) { throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); } diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketTagIndex.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketTagIndex.java new file mode 100644 index 00000000..069e42d2 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketTagIndex.java @@ -0,0 +1,80 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import java.io.IOException; + +import jakarta.servlet.http.HttpServletResponse; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.google.common.base.Strings; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.pspace.ifs.ksan.gw.data.DataPutBucketTagIndex; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.format.TagIndex; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; + +import org.slf4j.LoggerFactory; + +public class PutBucketTagIndex extends S3Request { + public PutBucketTagIndex(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(PutBucketTagIndex.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_PUT_BUCKET_TAG_INDEX_START); + String bucket = s3Parameter.getBucketName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + DataPutBucketTagIndex dataPutBucketTagIndex = new DataPutBucketTagIndex(s3Parameter); + dataPutBucketTagIndex.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_PUT_BUCKET_TAGGING, s3Parameter, dataPutBucketTagIndex)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + + String tagIndexXml = dataPutBucketTagIndex.getTagIndexXml(); + logger.info(GWConstants.LOG_PUT_BUCKET_TAG_INDEX, tagIndexXml); + + if (!Strings.isNullOrEmpty(tagIndexXml)) { + TagIndex tagIndex = new TagIndex(); + try { + tagIndex = new XmlMapper().readValue(tagIndexXml, TagIndex.class); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + if (tagIndex.status != null && tagIndex.status.equalsIgnoreCase(GWConstants.STATUS_ENABLED)) { + updateBucketTagIndex(bucket, true); + } else if (tagIndex.status != null && tagIndex.status.equalsIgnoreCase(GWConstants.STATUS_DISABLED)) { + updateBucketTagIndex(bucket, false); + } else { + throw new GWException(GWErrorCode.MALFORMED_X_M_L, s3Parameter); + } + } + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketTagging.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketTagging.java index 83a0c559..b05eb09d 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketTagging.java @@ -39,20 +39,19 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_TAGGING_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); - + DataPutBucketTagging dataPutBucketTagging = new DataPutBucketTagging(s3Parameter); dataPutBucketTagging.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_PUT_BUCKET_TAGGING, s3Parameter, dataPutBucketTagging)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } String taggingXml = dataPutBucketTagging.getTaggingXml(); logger.debug(GWConstants.LOG_PUT_BUCKET_TAGGING, taggingXml); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketVersioning.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketVersioning.java index 016b1f8c..0c81c3ec 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketVersioning.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketVersioning.java @@ -39,21 +39,20 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_VERSIONING_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); - + DataPutBucketVersioning dataPutBucketVersioning = new DataPutBucketVersioning(s3Parameter); dataPutBucketVersioning.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_BUCKET_VERSIONING, s3Parameter, dataPutBucketVersioning)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + String versionXml = dataPutBucketVersioning.getVersioningXml(); if (!Strings.isNullOrEmpty(versionXml)) { Versioning versioning = new Versioning(); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketWebsite.java b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketWebsite.java index 4d6b99db..58e28287 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutBucketWebsite.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutBucketWebsite.java @@ -35,21 +35,20 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_BUCKET_WEBSITE_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); - + DataPutBucketWebsite dataPutBucketWebsite = new DataPutBucketWebsite(s3Parameter); dataPutBucketWebsite.extract(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_BUCKET_WEBSITE, s3Parameter, dataPutBucketWebsite)) { + checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + String webXml = dataPutBucketWebsite.getWebsiteXml(); updateBucketWeb(bucket, webXml); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutObject.java b/core/src/com/pspace/ifs/ksan/gw/api/PutObject.java index 2608bc02..3dd1de3c 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutObject.java @@ -17,6 +17,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.dataformat.xml.XmlMapper; import com.google.common.base.Strings; @@ -48,9 +49,6 @@ import com.pspace.ifs.ksan.objmanager.ObjManagerException.ResourceNotFoundException; import org.slf4j.LoggerFactory; - - - public class PutObject extends S3Request { public PutObject(S3Parameter s3Parameter) { @@ -68,21 +66,20 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); logger.debug(GWConstants.LOG_BUCKET_OBJECT, bucket, object); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); - DataPutObject dataPutObject = new DataPutObject(s3Parameter); dataPutObject.extract(); + boolean effectPolicy = checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT, s3Parameter, dataPutObject); + if (!effectPolicy) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + S3Metadata s3Metadata = new S3Metadata(); String cacheControl = dataPutObject.getCacheControl(); @@ -197,7 +194,8 @@ public void process() throws GWException { dataPutObject.getGrantFullControl(), dataPutObject.getGrantReadAcp(), dataPutObject.getGrantWriteAcp(), - s3Parameter); + s3Parameter, + false); logger.debug(GWConstants.LOG_ACL, aclXml); String bucketEncryption = getBucketInfo().getEncryption(); logger.debug("bucket encryption : {}", bucketEncryption); @@ -326,12 +324,16 @@ public void process() throws GWException { s3Metadata.setLegalHold(dataPutObject.getObjectLockLegalHold()); } + // long dbStart = System.currentTimeMillis(); String versioningStatus = getBucketVersioning(bucket); String versionId = null; Metadata objMeta = null; + boolean isExist = false; + try { // check exist object objMeta = open(bucket, object); + isExist = true; if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { versionId = String.valueOf(System.nanoTime()); } else { @@ -339,6 +341,8 @@ public void process() throws GWException { } } catch (GWException e) { logger.info(e.getMessage()); + // reset error code + s3Parameter.setErrorCode(GWConstants.EMPTY_STRING); if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { versionId = String.valueOf(System.nanoTime()); objMeta = createLocal(diskpoolId, bucket, object, versionId); @@ -347,6 +351,17 @@ public void process() throws GWException { objMeta = createLocal(diskpoolId, bucket, object, versionId); } } + // long dbEnd = System.currentTimeMillis(); + // logger.error("put, db op : {}", dbEnd - dbStart); + + if (isExist && !effectPolicy) { + if (Strings.isNullOrEmpty(objMeta.getAcl())) { + objMeta.setAcl(GWUtils.makeOriginalXml(aclXml, s3Parameter)); + } + checkGrantObject(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + + s3Parameter.setVersionId(versionId); S3ObjectOperation objectOperation = new S3ObjectOperation(objMeta, s3Metadata, s3Parameter, versionId, encryption); S3Object s3Object = objectOperation.putObject(); @@ -366,6 +381,7 @@ public void process() throws GWException { ObjectMapper jsonMapper = new ObjectMapper(); String jsonmeta = ""; try { + // jsonMapper.setSerializationInclusion(Include.NON_NULL); jsonmeta = jsonMapper.writeValueAsString(s3Metadata); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); @@ -374,6 +390,7 @@ public void process() throws GWException { logger.debug(GWConstants.LOG_PUT_OBJECT_PRIMARY_DISK_ID, objMeta.getPrimaryDisk().getId()); try { + // logger.debug("taggingxml : {}", taggingxml); objMeta.set(s3Object.getEtag(), taggingxml, jsonmeta, aclXml, s3Object.getFileSize()); objMeta.setVersionId(versionId, GWConstants.OBJECT_TYPE_FILE, true); int result = insertObject(bucket, object, objMeta); @@ -383,6 +400,8 @@ public void process() throws GWException { throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } + // long fileEnd = System.currentTimeMillis(); + // logger.error("put file op : {}", fileEnd - dbEnd); s3Parameter.getResponse().addHeader(HttpHeaders.ETAG, GWUtils.maybeQuoteETag(s3Object.getEtag())); if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { s3Parameter.getResponse().addHeader(GWConstants.X_AMZ_VERSION_ID, s3Object.getVersionId()); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutObjectAcl.java b/core/src/com/pspace/ifs/ksan/gw/api/PutObjectAcl.java index c87c56f2..787c09cd 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutObjectAcl.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutObjectAcl.java @@ -43,10 +43,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); String object = s3Parameter.getObjectName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -57,6 +54,7 @@ public void process() throws GWException { dataPutObjectAcl.extract(); String versionId = dataPutObjectAcl.getVersionId(); + s3Parameter.setVersionId(versionId); Metadata objMeta = null; if (Strings.isNullOrEmpty(versionId)) { @@ -64,11 +62,17 @@ public void process() throws GWException { } else { objMeta = open(bucket, object, versionId); } - - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); - checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); - + if (Strings.isNullOrEmpty(versionId)) { + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT_ACL, s3Parameter, dataPutObjectAcl)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + } else { + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT_VERSION_ACL, s3Parameter, dataPutObjectAcl)) { + checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); + } + } + accessControlPolicy = new AccessControlPolicy(); accessControlPolicy.aclList = new AccessControlList(); accessControlPolicy.aclList.grants = new ArrayList(); @@ -88,7 +92,8 @@ public void process() throws GWException { dataPutObjectAcl.getGrantFullControl(), dataPutObjectAcl.getGrantReadAcp(), dataPutObjectAcl.getGrantWriteAcp(), - s3Parameter); + s3Parameter, + true); logger.debug(GWConstants.LOG_ACL, xml); objMeta.setAcl(xml); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutObjectLegalHold.java b/core/src/com/pspace/ifs/ksan/gw/api/PutObjectLegalHold.java new file mode 100644 index 00000000..2986338b --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutObjectLegalHold.java @@ -0,0 +1,135 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import java.io.IOException; + +import jakarta.servlet.http.HttpServletResponse; + +import com.google.common.base.Strings; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; + +import com.pspace.ifs.ksan.gw.data.DataPutObjectLegalHold; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.format.LegalHold; +import com.pspace.ifs.ksan.gw.format.ObjectLockConfiguration; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.objmanager.Metadata; + +import org.slf4j.LoggerFactory; + +public class PutObjectLegalHold extends S3Request { + + public PutObjectLegalHold(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(PutObjectLegalHold.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_PUT_OBJECT_LEGALHOLD_START); + String bucket = s3Parameter.getBucketName(); + String object = s3Parameter.getObjectName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + DataPutObjectLegalHold dataPutObjectLegalHold = new DataPutObjectLegalHold(s3Parameter); + dataPutObjectLegalHold.extract(); + + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT_LEGAL_HOLD, s3Parameter, dataPutObjectLegalHold)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + + String versionId = dataPutObjectLegalHold.getVersionId(); + + Metadata objMeta = null; + if (Strings.isNullOrEmpty(versionId)) { + objMeta = open(bucket, object); + versionId = objMeta.getVersionId(); + } else { + objMeta = open(bucket, object, versionId); + } + + // meta info + S3Metadata s3Metadata = null; + String meta = ""; + ObjectMapper objectMapper = new ObjectMapper(); + try { + logger.debug(GWConstants.LOG_META, objMeta.getMeta()); + s3Metadata = objectMapper.readValue(objMeta.getMeta(), S3Metadata.class); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + try { + String objectLock = getBucketInfo().getObjectLock(); + if (Strings.isNullOrEmpty(objectLock)) { + logger.info("bucket objectlock is null or empty."); + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); + } + ObjectLockConfiguration oc = new XmlMapper().readValue(objectLock, ObjectLockConfiguration.class); + if (!oc.objectLockEnabled.equals(GWConstants.STATUS_ENABLED)) { + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); + } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + String status; + LegalHold lh; + try { + lh = new XmlMapper().readValue(dataPutObjectLegalHold.getLegalHoldXml(), LegalHold.class); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + status = lh.status; + if (!Strings.isNullOrEmpty(status)) { + if (!status.equals(GWConstants.ON) && !status.equals(GWConstants.OFF)) { + throw new GWException(GWErrorCode.MALFORMED_X_M_L, s3Parameter); + } + } else { + throw new GWException(GWErrorCode.MALFORMED_X_M_L, s3Parameter); + } + + s3Metadata.setLegalHold(status); + try { + // objectMapper.setSerializationInclusion(Include.NON_NULL); + meta = objectMapper.writeValueAsString(s3Metadata); + logger.debug("meta : {}", meta); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + objMeta.setMeta(meta); + updateObjectMeta(objMeta); + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutObjectRetention.java b/core/src/com/pspace/ifs/ksan/gw/api/PutObjectRetention.java index 0f529bbb..214ce8d6 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutObjectRetention.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutObjectRetention.java @@ -10,13 +10,28 @@ */ package com.pspace.ifs.ksan.gw.api; +import java.io.IOException; + +import jakarta.servlet.http.HttpServletResponse; + +import com.google.common.base.Strings; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; + import com.pspace.ifs.ksan.gw.data.DataPutObjectRetention; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.format.Retention; +import com.pspace.ifs.ksan.gw.format.ObjectLockConfiguration; import com.pspace.ifs.ksan.gw.identity.S3Bucket; import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.objmanager.Metadata; import org.slf4j.LoggerFactory; @@ -30,11 +45,9 @@ public PutObjectRetention(S3Parameter s3Parameter) { public void process() throws GWException { logger.info(GWConstants.LOG_PUT_OBJECT_RETENTION_START); String bucket = s3Parameter.getBucketName(); + String object = s3Parameter.getObjectName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -43,10 +56,103 @@ public void process() throws GWException { DataPutObjectRetention dataPutObjectRetention = new DataPutObjectRetention(s3Parameter); dataPutObjectRetention.extract(); - String versionId = dataPutObjectRetention.getVersionId(); + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT_RETENTION, s3Parameter, dataPutObjectRetention)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + + Metadata objMeta = null; + if (Strings.isNullOrEmpty(versionId)) { + objMeta = open(bucket, object); + versionId = objMeta.getVersionId(); + } else { + objMeta = open(bucket, object, versionId); + } + + // meta info + S3Metadata s3Metadata = null; + String meta = ""; + ObjectMapper objectMapper = new ObjectMapper(); + try { + logger.debug(GWConstants.LOG_META, objMeta.getMeta()); + s3Metadata = objectMapper.readValue(objMeta.getMeta(), S3Metadata.class); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + try { + String objectLock = getBucketInfo().getObjectLock(); + if (Strings.isNullOrEmpty(objectLock)) { + logger.info("bucket objectlock is null or empty."); + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); + } + ObjectLockConfiguration oc = new XmlMapper().readValue(objectLock, ObjectLockConfiguration.class); + if (!oc.objectLockEnabled.equals(GWConstants.STATUS_ENABLED)) { + logger.info("bucket objectlock is not equals Enabled."); + throw new GWException(GWErrorCode.INVALID_REQUEST, s3Parameter); + } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + String mode; + String retainUntilDate; + try { + Retention rt = new XmlMapper().readValue(dataPutObjectRetention.getRetentionXml(), Retention.class); + mode = rt.mode; + retainUntilDate = rt.date; + + if (!Strings.isNullOrEmpty(mode)) { + if (!rt.mode.equals(GWConstants.GOVERNANCE) && !rt.mode.equals(GWConstants.COMPLIANCE)) { + throw new GWException(GWErrorCode.MALFORMED_X_M_L, s3Parameter); + } + } else { + throw new GWException(GWErrorCode.MALFORMED_X_M_L, s3Parameter); + } + + if (!Strings.isNullOrEmpty(s3Metadata.getLockMode())) { + if (s3Metadata.getLockMode().equals(GWConstants.COMPLIANCE) && mode.equals(GWConstants.GOVERNANCE)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } + + if (!Strings.isNullOrEmpty(s3Metadata.getLockExpires())) { + long curDate = GWUtils.parseRetentionTimeExpire(s3Metadata.getLockExpires(), s3Parameter); + long newDate = GWUtils.parseRetentionTimeExpire(retainUntilDate, s3Parameter); + logger.info(GWConstants.LOG_CUR_NEW_DATE, curDate, newDate); + if (!Strings.isNullOrEmpty(dataPutObjectRetention.getBypassGovernanceRetention())) { + if (!dataPutObjectRetention.getBypassGovernanceRetention().equalsIgnoreCase(GWConstants.XML_TRUE) && curDate > newDate) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } else { + if (curDate > newDate) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } + } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + s3Metadata.setLockMode(mode); + s3Metadata.setLockExpires(retainUntilDate); + + try { + // objectMapper.setSerializationInclusion(Include.NON_NULL); + meta = objectMapper.writeValueAsString(s3Metadata); + logger.debug("meta : {}", meta); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + objMeta.setMeta(meta); + updateObjectMeta(objMeta); - throw new GWException(GWErrorCode.NOT_IMPLEMENTED, s3Parameter); + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); } } diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutObjectTagging.java b/core/src/com/pspace/ifs/ksan/gw/api/PutObjectTagging.java index a022c48a..31fb5ef5 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutObjectTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutObjectTagging.java @@ -15,6 +15,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.dataformat.xml.XmlMapper; import com.google.common.base.Strings; @@ -44,23 +45,37 @@ public void process() throws GWException { logger.info(GWConstants.LOG_PUT_OBJECT_TAGGING_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } - - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); String object = s3Parameter.getObjectName(); DataPutObjectTagging dataPutObjectTagging = new DataPutObjectTagging(s3Parameter); dataPutObjectTagging.extract(); + String versionId = dataPutObjectTagging.getVersionId(); + + Metadata objMeta = null; + if (Strings.isNullOrEmpty(versionId)) { + objMeta = open(bucket, object); + } else { + objMeta = open(bucket, object, versionId); + } + s3Parameter.setTaggingInfo(objMeta.getTag()); + if (Strings.isNullOrEmpty(versionId)) { + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT_TAGGING, s3Parameter, dataPutObjectTagging)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } else { + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT_VERSION_TAGGING, s3Parameter, dataPutObjectTagging)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } + String taggingCount = GWConstants.TAGGING_INIT; String taggingXml = dataPutObjectTagging.getTaggingXml(); try { @@ -95,14 +110,6 @@ public void process() throws GWException { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } - - String versionId = dataPutObjectTagging.getVersionId(); - Metadata objMeta = null; - if (Strings.isNullOrEmpty(versionId)) { - objMeta = open(bucket, object); - } else { - objMeta = open(bucket, object, versionId); - } S3Metadata s3Metadata = null; ObjectMapper objectMapper = new ObjectMapper(); @@ -115,10 +122,11 @@ public void process() throws GWException { } s3Metadata.setTaggingCount(taggingCount); - ObjectMapper jsonMapper = new ObjectMapper(); + // ObjectMapper jsonMapper = new ObjectMapper(); String jsonMeta = ""; try { - jsonMeta = jsonMapper.writeValueAsString(s3Metadata); + objectMapper.setSerializationInclusion(Include.NON_NULL); + jsonMeta = objectMapper.writeValueAsString(s3Metadata); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/PutPublicAccessBlock.java b/core/src/com/pspace/ifs/ksan/gw/api/PutPublicAccessBlock.java index 207c75ff..4c54e89d 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/PutPublicAccessBlock.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/PutPublicAccessBlock.java @@ -16,6 +16,7 @@ import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; import org.slf4j.LoggerFactory; diff --git a/core/src/com/pspace/ifs/ksan/gw/api/RestoreObject.java b/core/src/com/pspace/ifs/ksan/gw/api/RestoreObject.java new file mode 100644 index 00000000..88c99e69 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/RestoreObject.java @@ -0,0 +1,78 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.api; + +import jakarta.servlet.http.HttpServletResponse; + +import com.pspace.ifs.ksan.gw.data.DataPutPublicAccessBlock; +import com.pspace.ifs.ksan.gw.data.DataRestoreObject; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.google.common.base.Strings; +import org.slf4j.LoggerFactory; + +public class RestoreObject extends S3Request { + + public RestoreObject(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(PutPublicAccessBlock.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_RESTORE_OBJECT_START); + String bucket = s3Parameter.getBucketName(); + String object = s3Parameter.getObjectName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + + DataRestoreObject dataRestoreObject = new DataRestoreObject(s3Parameter); + dataRestoreObject.extract(); + String versionId = dataRestoreObject.getVersionId(); + String restoreXml = dataRestoreObject.getRetoreXml(); + if (Strings.isNullOrEmpty(versionId)) { + versionId = GWConstants.VERSIONING_DISABLE_TAIL; + } + logger.info("RestoreRequest : {}", restoreXml); + + Metadata objMeta = null; + if (Strings.isNullOrEmpty(versionId)) { + objMeta = open(bucket, object); + } else { + objMeta = open(bucket, object, versionId); + } + + s3Parameter.setTaggingInfo(objMeta.getTag()); + if (Strings.isNullOrEmpty(versionId)) { + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT_TAGGING, s3Parameter, dataRestoreObject)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } else { + if (!checkPolicyBucket(GWConstants.ACTION_PUT_OBJECT_VERSION_TAGGING, s3Parameter, dataRestoreObject)) { + checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + } + } + + insertRestoreObject(bucket, object, versionId, restoreXml); + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_ACCEPTED); + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/api/S3Request.java b/core/src/com/pspace/ifs/ksan/gw/api/S3Request.java index 5847fe94..7a8d9004 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/S3Request.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/S3Request.java @@ -14,13 +14,17 @@ import java.util.Date; import java.util.List; import java.util.TimeZone; +import java.util.Map; import javax.xml.stream.XMLStreamException; import javax.xml.stream.XMLStreamWriter; - +import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.ArrayNode; import com.fasterxml.jackson.dataformat.xml.XmlMapper; + import com.google.common.base.Strings; import com.google.common.hash.HashFunction; import com.google.common.hash.Hashing; @@ -30,6 +34,7 @@ import com.pspace.ifs.ksan.gw.format.AccessControlPolicy.AccessControlList.Grant; import com.pspace.ifs.ksan.libs.identity.ObjectListParameter; import com.pspace.ifs.ksan.libs.identity.S3BucketSimpleInfo; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; import com.pspace.ifs.ksan.libs.identity.S3ObjectList; import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.gw.object.objmanager.ObjManagerHelper; @@ -42,6 +47,12 @@ import com.pspace.ifs.ksan.objmanager.ObjManagerException.ResourceAlreadyExistException; import com.pspace.ifs.ksan.objmanager.ObjManagerException.ResourceNotFoundException; import com.pspace.ifs.ksan.objmanager.ObjMultipart; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.data.S3DataRequest; +import com.pspace.ifs.ksan.gw.format.Policy; +import com.pspace.ifs.ksan.gw.format.Policy.Statement; +import com.pspace.ifs.ksan.gw.condition.PolicyCondition; +import com.pspace.ifs.ksan.gw.condition.PolicyConditionFactory; import org.slf4j.Logger; @@ -143,6 +154,15 @@ protected void initBucketInfo(String bucket) throws GWException { logger.info(GWConstants.LOG_BUCKET_IS_NOT_EXIST, bucket); throw new GWException(GWErrorCode.NO_SUCH_BUCKET, s3Parameter); } + + S3Bucket s3Bucket = new S3Bucket(); + s3Bucket.setBucket(bucket); + s3Bucket.setUserName(dstBucket.getUserName()); + s3Bucket.setCors(dstBucket.getCors()); + s3Bucket.setAccess(dstBucket.getAccess()); + s3Bucket.setPolicy(dstBucket.getPolicy()); + logger.info("bucket policy : {}", dstBucket.getPolicy()); + s3Parameter.setBucket(s3Bucket); } protected Bucket getSomeBucket(String bucket) throws GWException { @@ -620,7 +640,7 @@ protected boolean isGrantObjectOwner(Metadata meta, String id, String s3grant) t } } - if (accessControlPolicyObject.aclList != null) { + if (accessControlPolicyObject.aclList == null) { return false; } @@ -750,6 +770,12 @@ protected Metadata open(String bucket, String object) throws GWException { try { setObjManager(); meta = objManager.open(bucket, object); + + if (!Strings.isNullOrEmpty(meta.getAcl())) { + meta.setAcl(GWUtils.makeOriginalXml(meta.getAcl(), s3Parameter)); + } else { + logger.debug("acl is null or empty."); + } } catch (ResourceNotFoundException e) { throw new GWException(GWErrorCode.NO_SUCH_KEY, s3Parameter); } catch (Exception e) { @@ -772,8 +798,13 @@ protected Metadata open(String bucket, String objcet, String versionId) throws G try { setObjManager(); meta = objManager.open(bucket, objcet, versionId); + + if (!Strings.isNullOrEmpty(meta.getAcl())) { + meta.setAcl(GWUtils.makeOriginalXml(meta.getAcl(), s3Parameter)); + } else { + logger.debug("acl is null or empty."); + } } catch (ResourceNotFoundException e) { - PrintStack.logging(logger, e); throw new GWException(GWErrorCode.NO_SUCH_KEY, s3Parameter); } catch (Exception e) { PrintStack.logging(logger, e); @@ -1036,8 +1067,13 @@ protected int insertObject(String bucket, String object, Metadata data) throws G setObjManager(); result = objManager.close(bucket, object, data); } catch (Exception e) { - PrintStack.logging(logger, e); - throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + // duplicate key error + try { + result = objManager.close(bucket, object, data); + } catch (Exception e1) { + PrintStack.logging(logger, e1); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } } finally { try { releaseObjManager(); @@ -1152,6 +1188,47 @@ protected void updateBucketTagging(String bucket, String tagging) throws GWExcep } } + protected boolean isBucketTagIndex(String bucket) throws GWException { + boolean enable = false; + try { + setObjManager(); + enable = objManager.getObjectTagsIndexing().isIndexingEnabled(bucket); + } catch (Exception e) { + logger.error(e.getMessage()); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } + + return enable; + } + + protected void updateBucketTagIndex(String bucket, boolean enable) throws GWException { + try { + setObjManager(); + if (enable) { + objManager.getObjectTagsIndexing().enableIndexing(bucket); + } else { + objManager.getObjectTagsIndexing().disableIndexing(bucket); + } + } catch (Exception e) { + logger.error(e.getMessage()); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } + } + protected void updateBucketWeb(String bucket, String web) throws GWException { try { setObjManager(); @@ -1237,6 +1314,23 @@ protected void updateBucketPolicy(String bucket, String policy) throws GWExcepti } } + protected void updateBucketLogging(String bucket, String logging) throws GWException { + try { + setObjManager(); + objManager.updateBucketLogging(bucket, logging); + } catch (Exception e) { + logger.error(e.getMessage()); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } + } + protected Bucket getBucket(String bucket) throws GWException { Bucket bucketInfo = null; try { @@ -1279,6 +1373,26 @@ protected void putBucketVersioning(String bucket, String status) throws GWExcept } } + protected void insertRestoreObject(String bucket, String object, String versionId, String restoreXml) throws GWException { + try { + setObjManager(); + objManager.getRestoreObjects().insertRequest(bucket, object, versionId, restoreXml); + } catch (ResourceNotFoundException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.NO_SUCH_KEY, s3Parameter); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } + } + protected List listBucketSimpleInfo(String userName, String userId) throws GWException { List bucketList = null; try { @@ -1299,6 +1413,26 @@ protected List listBucketSimpleInfo(String userName, String return bucketList; } + protected List listBucketTags(String bucket, String tags, int max) throws GWException { + List tagList = null; + try { + setObjManager(); + tagList = objManager.getObjectTagsIndexing().getObjectWithTags(bucket, tags, max); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } + + return tagList; + } + protected ObjectListParameter listObject(String bucket, S3ObjectList s3ObjectList) throws GWException { ObjectListParameter objectListParameter = null; try { @@ -1393,6 +1527,23 @@ protected void updateObjectAcl(Metadata meta) throws GWException { } } + protected void updateObjectRestore(Metadata meta) throws GWException { + try { + setObjManager(); + objManager.updateObjectTagging(meta); + } catch(Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } + } + protected ObjMultipart getInstanceObjMultipart(String bucket) throws GWException { ObjMultipart objMultipart = null; try { @@ -1498,7 +1649,7 @@ protected void checkGrantObjectOwner(boolean pub, Metadata meta, String uid, Str protected void checkGrantObject(boolean pub, Metadata meta, String uid, String grant) throws GWException { if (pub) { - if (!isGrantObject(meta,GWConstants.LOG_REQUEST_ROOT_ID, grant)) { + if (!isGrantObject(meta, GWConstants.LOG_REQUEST_ROOT_ID, grant)) { logger.error(GWConstants.LOG_REQUEST_PUBLIC_ACCESS_DENIED); throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } @@ -1509,4 +1660,520 @@ protected void checkGrantObject(boolean pub, Metadata meta, String uid, String g } } } + + protected boolean checkPolicyBucket(String action, S3DataRequest dataRequest) throws GWException { + boolean effect = false; + String policyJson = s3Parameter.getBucket().getPolicy(); + if (Strings.isNullOrEmpty(policyJson)) { + return effect; + } + + Policy policy = null; + // read policy + ObjectMapper jsonMapper = new ObjectMapper(); + try { + policy = jsonMapper.readValue(policyJson, Policy.class); + + if (policy == null) { + return effect; + } + } catch (JsonMappingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + // check policy - loop statement + for (Statement s : policy.statements) { + + // action check + for (String a : s.actions) { + boolean effectcheck = false; + if (action.equals(a) || GWConstants.ACTION_ALL.equals(a)) { + // check principal (id) + for (String aws : s.principal.aws) { + if (policyIdCheck(aws)) + break; + } + + // check Resource (object path, bucket path) + for (String resource : s.resources) { + if (policyResourceCheck(resource)) + break; + } + + if (conditioncheck(s3Parameter, s, dataRequest)) { + effectcheck = true; + } + + if (s.effect.equals(GWConstants.ALLOW)) { + if (!effectcheck) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } else { + effect = true; + } + } else if (s.effect.equals(GWConstants.DENY) && effectcheck) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } + } + } + + return effect; + } + + protected boolean policyIdCheck(String aws) { + boolean effectcheck = false; + if (aws.equals(GWConstants.ASTERISK)) { + effectcheck = true; + return effectcheck; + } + + String[] arn = aws.split(GWConstants.COLON, -1); + if (!Strings.isNullOrEmpty(arn[4])) { + if (s3Parameter.getUser().getUserId().equals(arn[4])) { + effectcheck = true; + return effectcheck; + } + } + + return effectcheck; + } + + protected boolean policyResourceCheck(String resource) { + boolean effectcheck = false; + + String[] res = resource.split(GWConstants.COLON, -1); + if (Strings.isNullOrEmpty(res[5])) { + return effectcheck; + } + + // all resource check + if (res[5].equals("*")) { + effectcheck = true; + return effectcheck; + } + + // bucket resource check + String[] path = res[5].split(GWConstants.SLASH, 2); + if (path.length == 1) { + if (path[0].equals(s3Parameter.getBucketName())) { + effectcheck = true; + return effectcheck; + } + return effectcheck; + } + + // object resource check + if (path.length == 2) { + if (!path[0].equals(s3Parameter.getBucketName())) { + return effectcheck; + } + + if (path[1].equals(GWConstants.ASTERISK)) { + effectcheck = true; + return effectcheck; + } + + if (GWUtils.likematch(path[1], s3Parameter.getObjectName())) { + effectcheck = true; + return effectcheck; + } + } + + return effectcheck; + } + + protected boolean conditioncheck(S3Parameter s3Parameter, Statement s, S3DataRequest dataRequest) throws GWException { + // condition check + boolean conditioncheck = true; + if (s.condition == null) + return conditioncheck; + + for (Map.Entry entry : s.condition.getUserExtensions().entries()) { + PolicyCondition pc = PolicyConditionFactory.createPolicyCondition(entry.getKey(), entry.getValue()); + if (pc == null) { + continue; + } + + pc.process(); + + String comp = null; + switch (pc.getKey()) { + case GWConstants.KEY_AUTH_TYPE: + break; + case GWConstants.KEY_DELIMITER: + comp = dataRequest.getPolicyDelimter(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_MAXKYES: + comp = dataRequest.getPolicyMaxkeys(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_PREFIX: + comp = dataRequest.getPolicyPrefix(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_CONTENT_SHA256: + comp = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_CONTENT_SHA256); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_ACL: + comp = dataRequest.getXAmzAcl(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_COPY_SOURCE: + comp = dataRequest.getXAmzCopySource(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_GRANT_FULL_CONTROL: + comp = dataRequest.getXAmzGrantFullControl(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_XAMZ_GRANT_READ: + comp = dataRequest.getXAmzGrantRead(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_GRANT_READ_ACP: + comp = dataRequest.getXAmzGrantReadAcp(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_GRANT_WRITE: + comp = dataRequest.getXAmzGrantWrite(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_GRANT_WRITE_ACP: + comp = dataRequest.getXAmzGrantWriteAcp(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_METADATA_DIRECTIVE: + comp = dataRequest.getXAmzMetadataDirective(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_SERVER_SIDE_ENCRYPTION: + comp = dataRequest.getXAmzServerSideEncryption(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID: + comp = dataRequest.getXAmzServerSideEncryptionAwsKmsKeyId(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_STORAGE_CLASS: + comp = dataRequest.getXAmzStorageClass(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_WEBSITE_REDIRECT_LOCATION: + comp = dataRequest.getXAmzWebsiteRedirectLocation(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_OBJECT_LOCK_MODE: + comp = dataRequest.getXAmzObjectLockMode(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE: + comp = dataRequest.getXAmzObjectLockRetainUntilDate(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_OBJECT_LOCK_REMAINING_RETENTION_DAYS: + comp = dataRequest.getXAmzObjectLockRemainingRetentionDays(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_OBJECT_LOCK_LEGAL_HOLD: + comp = dataRequest.getXAmzObjectLockLegalHold(); + conditioncheck = pc.compare(comp); + break; + default: + break; + } + + if (pc.getKey().startsWith(GWConstants.KEY_EXISTING_OBJECT_TAG)) { + conditioncheck = pc.compareTagging(s3Parameter); + } + + if (conditioncheck == false) { + break; + } + } + + return conditioncheck; + } + + public void retentionCheck(String meta, String bypassGovernanceRetention, S3Parameter s3Parameter) throws GWException { + S3Metadata retentionMetadata = null; + if (!Strings.isNullOrEmpty(meta)) { + try { + retentionMetadata = new ObjectMapper().readValue(meta, S3Metadata.class); + if (!Strings.isNullOrEmpty(retentionMetadata.getLockMode()) && retentionMetadata.getLockMode().equalsIgnoreCase(GWConstants.GOVERNANCE)) { + if (Strings.isNullOrEmpty(bypassGovernanceRetention) + || (!Strings.isNullOrEmpty(bypassGovernanceRetention) && !bypassGovernanceRetention.equalsIgnoreCase(GWConstants.STRING_TRUE))) { + // check retention date + long untilDate = GWUtils.parseRetentionTimeExpire(retentionMetadata.getLockExpires(), s3Parameter); + long now = System.currentTimeMillis() / 1000; + if (untilDate > now) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } + } + + if (!Strings.isNullOrEmpty(retentionMetadata.getLockMode()) && retentionMetadata.getLockMode().equalsIgnoreCase(GWConstants.COMPLIANCE)) { + // check retention date + long untilDate = GWUtils.parseRetentionTimeExpire(retentionMetadata.getLockExpires(), s3Parameter); + long now = System.currentTimeMillis() / 1000; + if (untilDate > now) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } + + if (!Strings.isNullOrEmpty(retentionMetadata.getLegalHold()) && retentionMetadata.getLegalHold().equalsIgnoreCase(GWConstants.ON)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } + } + + public boolean checkPolicyBucket(String givenAction, S3Parameter s3Parameter, S3DataRequest dataRequest) throws GWException { + logger.info("checkPolicyBucket..."); + boolean effect = false; + String policyJson = s3Parameter.getBucket().getPolicy(); + if (Strings.isNullOrEmpty(policyJson)) { + return effect; + } + logger.info("bucket policy : {}", policyJson); + Policy policy = null; + ObjectMapper jsonMapper = new ObjectMapper(); + try { + policy = jsonMapper.readValue(policyJson, Policy.class); + if (policy == null) { + return effect; + } + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + logger.info("1..."); + // check policy + for (Statement statement : policy.statements) { + // check action + for (String action : statement.actions) { + boolean isEffect = false; + logger.info("given action : {}, action : {}", givenAction, action); + if (givenAction.equals(action) || GWConstants.ACTION_ALL.equals(action)) { + // check principal (id) + for (String aws : statement.principal.aws) { + if (checkPolicyId(aws, s3Parameter)) { + break; + } + } + logger.info("2..."); + // check Resource (object path, bucket path) + for (String resource : statement.resources) { + if (checkPolicyResource(resource, s3Parameter)) { + break; + } + } + logger.info("3..."); + if (checkCondition(s3Parameter, statement, dataRequest)) { + isEffect = true; + } + logger.info("4..."); + if (statement.effect.equals(GWConstants.ALLOW)) { + logger.info("allow... {}", isEffect); + if (isEffect) { + effect = true; + } else { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } else if (statement.effect.equals(GWConstants.DENY)) { + if (isEffect) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + } + } + } + } + + return effect; + } + + public boolean checkPolicyId(String aws, S3Parameter s3Parameter) { + boolean effectcheck = false; + if (aws.equals(GWConstants.ASTERISK)) { + effectcheck = true; + return effectcheck; + } + + String[] arn = aws.split(GWConstants.COLON, -1); + if (!Strings.isNullOrEmpty(arn[4])) { + if (s3Parameter.getUser().getUserId().equals(arn[4])) { + effectcheck = true; + return effectcheck; + } + } + + return effectcheck; + } + + public boolean checkPolicyResource(String resource, S3Parameter s3Parameter) { + boolean effectcheck = false; + + String[] res = resource.split(GWConstants.COLON, -1); + if (Strings.isNullOrEmpty(res[5])) { + return effectcheck; + } + + // all resource check + if (res[5].equals(GWConstants.ASTERISK)) { + effectcheck = true; + return effectcheck; + } + + // bucket resource check + String[] path = res[5].split(GWConstants.SLASH, 2); + if (path.length == 1) { + if (path[0].equals(s3Parameter.getBucketName())) { + effectcheck = true; + return effectcheck; + } + return effectcheck; + } + + // object resource check + if (path.length == 2) { + if (!path[0].equals(s3Parameter.getBucketName())) { + return effectcheck; + } + + if (path[1].equals(GWConstants.ASTERISK)) { + effectcheck = true; + return effectcheck; + } + + if (GWUtils.likematch(path[1], s3Parameter.getObjectName())) { + effectcheck = true; + return effectcheck; + } + } + + return effectcheck; + } + + public boolean checkCondition(S3Parameter s3Parameter, Statement s, S3DataRequest dataRequest) throws GWException { + // condition check + boolean conditioncheck = true; + if (s.condition == null) + return conditioncheck; + + for (Map.Entry entry : s.condition.getUserExtensions().entries()) { + PolicyCondition pc = PolicyConditionFactory.createPolicyCondition(entry.getKey(), entry.getValue()); + if (pc == null) { + continue; + } + + pc.process(); + + String comp = null; + logger.info("PolicyCondition key : {}", pc.getKey()); + switch (pc.getKey()) { + case GWConstants.KEY_AUTH_TYPE: + break; + case GWConstants.KEY_DELIMITER: + comp = dataRequest.getPolicyDelimter(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_MAXKYES: + comp = dataRequest.getPolicyMaxkeys(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_PREFIX: + comp = dataRequest.getPolicyPrefix(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_CONTENT_SHA256: + comp = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_CONTENT_SHA256); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_ACL: + comp = dataRequest.getXAmzAcl(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_COPY_SOURCE: + comp = dataRequest.getXAmzCopySource(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_GRANT_FULL_CONTROL: + comp = dataRequest.getXAmzGrantFullControl(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_XAMZ_GRANT_READ: + comp = dataRequest.getXAmzGrantRead(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_GRANT_READ_ACP: + comp = dataRequest.getXAmzGrantReadAcp(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_GRANT_WRITE: + comp = dataRequest.getXAmzGrantWrite(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_GRANT_WRITE_ACP: + comp = dataRequest.getXAmzGrantWriteAcp(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_METADATA_DIRECTIVE: + comp = dataRequest.getXAmzMetadataDirective(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_SERVER_SIDE_ENCRYPTION: + comp = dataRequest.getXAmzServerSideEncryption(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID: + comp = dataRequest.getXAmzServerSideEncryptionAwsKmsKeyId(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_STORAGE_CLASS: + comp = dataRequest.getXAmzStorageClass(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_WEBSITE_REDIRECT_LOCATION: + comp = dataRequest.getXAmzWebsiteRedirectLocation(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_OBJECT_LOCK_MODE: + comp = dataRequest.getXAmzObjectLockMode(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE: + comp = dataRequest.getXAmzObjectLockRetainUntilDate(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_OBJECT_LOCK_REMAINING_RETENTION_DAYS: + comp = dataRequest.getXAmzObjectLockRemainingRetentionDays(); + conditioncheck = pc.compare(comp); + break; + case GWConstants.KEY_X_AMZ_OBJECT_LOCK_LEGAL_HOLD: + comp = dataRequest.getXAmzObjectLockLegalHold(); + conditioncheck = pc.compare(comp); + break; + default: + break; + } + + if (pc.getKey().startsWith(GWConstants.KEY_EXISTING_OBJECT_TAG)) { + conditioncheck = pc.compareTagging(s3Parameter); + } + + if (conditioncheck == false) { + break; + } + } + + return conditioncheck; + } } diff --git a/core/src/com/pspace/ifs/ksan/gw/api/S3RequestFactory.java b/core/src/com/pspace/ifs/ksan/gw/api/S3RequestFactory.java index e1541d59..51d7a514 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/S3RequestFactory.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/S3RequestFactory.java @@ -38,6 +38,7 @@ public class S3RequestFactory { private final String OP_DELETE_OBJECTLOCK = "REST.DELETE.OBJECTLOCK"; private final String OP_DELETE_NOTIFICATION = "REST.DELETE.NOTIFICATION"; private final String OP_DELETE_REPLICATION = "REST.DELETE.REPLICATION"; + private final String OP_DELETE_BUCKET_TAG_INDEX = "REST.DELETE.BUCKET.TAG_INDEX"; private final String OP_GET_LISTBUCKET = "REST.GET.LISTBUCKET"; private final String OP_GET_WEBSITE = "REST.GET.WEBSITE"; @@ -51,6 +52,7 @@ public class S3RequestFactory { private final String OP_GET_NOTIFICATION = "REST.GET.NOTIFICATION"; private final String OP_GET_BUCKET_POLICY_STATUS = "REST.GET.BUCKET.POLICY.STATUS"; private final String OP_GET_REPLICATION = "REST.GET.REPLICATION"; + private final String OP_GET_BUCKET_TAG_INDEX = "REST.GET.BUCKET.TAG_INDEX"; private final String OP_GET_ENCRYPTION = "REST.GET.ENCRYPTION"; private final String OP_GET_BUCKET_ACL = "REST.GET.BUCKET.ACL"; @@ -60,6 +62,7 @@ public class S3RequestFactory { private final String OP_GET_LISTOBJECTSV2 = "REST.GET.LISTOBJECTV2"; private final String OP_GET_LISTOBJECTS = "REST.GET.LISTOBJECT"; private final String OP_GET_LISTVERSIONS = "REST.GET.LISTVERSIONS"; + private final String OP_GET_LIST_TAG_SEARCH = "REST.GET.LISTTAGSEARCH"; private final String OP_GET_OBJECT_ACL = "REST.GET.OBJECT.ACL"; private final String OP_GET_OBJECT_RETENTION = "REST.GET.OBJECT.RETENTION"; private final String OP_GET_OBJECT_LEGAL_HOLD = "REST.GET.OBJECT.LEGAL.HOLD"; @@ -74,6 +77,7 @@ public class S3RequestFactory { private final String OP_POST_UPLOAD = "REST.POST.UPLOAD"; private final String OP_POST_COMPLETE = "REST.POST.COMPLETEUPLOAD"; private final String OP_POST_OBJECT = "REST.POST.OBJECT"; + private final String OP_POST_RESTOREOBJECT = "REST.POST.RESTOREOBJECT"; private final String OP_PUT_WEBSITE = "REST.PUT.WEBSITE"; private final String OP_PUT_POLICY = "REST.PUT.POLICY"; @@ -89,6 +93,7 @@ public class S3RequestFactory { private final String OP_PUT_OBJECTLOCK = "REST.PUT.OBJECTLOCK"; private final String OP_PUT_NOTIFICATION = "REST.PUT.NOTIFICATION"; private final String OP_PUT_REPLICATION = "REST.PUT.REPLICATION"; + private final String OP_PUT_BUCKET_TAG_INDEX = "REST.PUT.BUCKET.TAG_INDEX"; private final String OP_PUT_OBJECT_PART_COPY = "REST.PUT.OBJECT.PART.COPY"; private final String OP_PUT_OBJECT_PART = "REST.PUT.OBJECT.PART"; @@ -101,6 +106,7 @@ public class S3RequestFactory { private final String OP_OPTIONS = "REST.OPTIONS"; + private final String OP_ADMIN_GET_BUCKET_ACL = "ADMIN.GET.BUCKET.ACL"; private final String OP_ADMIN_DELETE_OBJECT = "ADMIN.DELETE.OBJECT"; private final String OP_ADMIN_DELETE_OBJECT_TAGGING = "ADMIN.DELETE.OBJECT.TAGGING"; private final String OP_ADMIN_DELETE_OBJECT_UPLOAD = "ADMIN.DELETE.OBJECT.UPLOAD"; @@ -145,12 +151,18 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_ENCRYPTION))) { s3Parameter.setOperation(OP_DELETE_ENCRYPTION); return new DeleteBucketEncryption(s3Parameter); + } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_LOGGING))) { + s3Parameter.setOperation(OP_DELETE_LOGGING); + return new DeleteBucketLogging(s3Parameter); } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_OBJECT_LOCK))) { s3Parameter.setOperation(OP_DELETE_OBJECTLOCK); return new DeleteBucketObjectLock(s3Parameter); } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_REPLICATION))) { s3Parameter.setOperation(OP_DELETE_REPLICATION); return new DeleteBucketReplication(s3Parameter); + } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_TAG_INDEX))) { + s3Parameter.setOperation(OP_DELETE_BUCKET_TAG_INDEX); + return new DeleteBucketTagIndex(s3Parameter); } if (s3Parameter.isPublicAccess()) { @@ -223,9 +235,17 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_ENCRYPTION))) { s3Parameter.setOperation(OP_GET_ENCRYPTION); return new GetBucketEncryption(s3Parameter); + } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_LOGGING))) { + s3Parameter.setOperation(OP_GET_LOGGING); + return new GetBucketLogging(s3Parameter); } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_ACL))) { - s3Parameter.setOperation(OP_GET_BUCKET_ACL); - return new GetBucketAcl(s3Parameter); + if (s3Parameter.isAdmin()) { + s3Parameter.setOperation(OP_ADMIN_GET_BUCKET_ACL); + return new KsanGetBucketAcl(s3Parameter); + } else { + s3Parameter.setOperation(OP_GET_BUCKET_ACL); + return new GetBucketAcl(s3Parameter); + } } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_LOCATION))) { s3Parameter.setOperation(OP_GET_LOCATION); return new GetBucketLocation(s3Parameter); @@ -237,13 +257,16 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { return new GetBucketVersioning(s3Parameter); } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_OBJECT_LOCK))) { s3Parameter.setOperation(OP_GET_OBJECTLOCK); - return new GetObjectLockConfiguration(s3Parameter); + return new GetBucketObjectLock(s3Parameter); } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_REPLICATION))) { s3Parameter.setOperation(OP_GET_REPLICATION); return new GetBucketReplication(s3Parameter); } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_POLICY_STATUS))) { s3Parameter.setOperation(OP_GET_BUCKET_POLICY_STATUS); return new GetBucketPolicyStatus(s3Parameter); + } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_TAG_INDEX))) { + s3Parameter.setOperation(OP_GET_BUCKET_TAG_INDEX); + return new GetBucketTagIndex(s3Parameter); } if (s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_LIST_TYPE) != null) { @@ -252,6 +275,9 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { } else if (s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_VERSIONS) != null) { s3Parameter.setOperation(OP_GET_LISTVERSIONS); return new ListObjectVersions(s3Parameter); + } else if (s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_LIST_TAG_SEARCH) != null) { + s3Parameter.setOperation(OP_GET_LIST_TAG_SEARCH); + return new ListBucketTagSearch(s3Parameter); } else { s3Parameter.setOperation(OP_GET_LISTOBJECTS); return new ListObjects(s3Parameter); @@ -270,6 +296,9 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_RETENTION))) { s3Parameter.setOperation(OP_GET_OBJECT_RETENTION); return new GetObjectRetention(s3Parameter); + } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_LEGAL_HOLD))) { + s3Parameter.setOperation(OP_GET_OBJECT_LEGAL_HOLD); + return new GetObjectLegalHold(s3Parameter); } else if (s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_UPLOAD_ID) != null) { s3Parameter.setOperation(OP_GET_OBJECT_LISTPARTS); return new ListParts(s3Parameter); @@ -300,8 +329,13 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { } if (GWConstants.CATEGORY_OBJECT.equals(s3Parameter.getPathCategory())) { - s3Parameter.setOperation(OP_HEAD_OBJECT); - return new HeadObject(s3Parameter); + if (s3Parameter.isAdmin()) { + s3Parameter.setOperation(OP_ADMIN_HEAD_OBJECT); + return new KsanHeadObject(s3Parameter); + } else { + s3Parameter.setOperation(OP_HEAD_OBJECT); + return new HeadObject(s3Parameter); + } } break; @@ -326,6 +360,9 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { s3Parameter.setOperation(OP_POST_COMPLETE); return new CompleteMultipartUpload(s3Parameter); } + } else if (s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_RESTORE) != null) { + s3Parameter.setOperation(OP_POST_RESTOREOBJECT); + return new RestoreObject(s3Parameter); } if (s3Parameter.getRequest().getHeader(HttpHeaders.CONTENT_TYPE) != null && @@ -358,6 +395,9 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_ENCRYPTION))) { s3Parameter.setOperation(OP_PUT_ENCRYPTION); return new PutBucketEncryption(s3Parameter); + } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_LOGGING))) { + s3Parameter.setOperation(OP_PUT_LOGGING); + return new PutBucketLogging(s3Parameter); } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_ACL))) { s3Parameter.setOperation(OP_PUT_BUCKET_ACL); return new PutBucketAcl(s3Parameter); @@ -366,10 +406,13 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { return new PutBucketVersioning(s3Parameter); } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_OBJECT_LOCK))) { s3Parameter.setOperation(OP_PUT_OBJECTLOCK); - return new PutObjectLockConfiguration(s3Parameter); + return new PutBucketObjectLock(s3Parameter); } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_REPLICATION))) { s3Parameter.setOperation(OP_PUT_REPLICATION); return new PutBucketReplication(s3Parameter); + } else if (GWConstants.EMPTY_STRING.equals(s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_TAG_INDEX))) { + s3Parameter.setOperation(OP_PUT_BUCKET_TAG_INDEX); + return new PutBucketTagIndex(s3Parameter); } if (s3Parameter.isPublicAccess()) { @@ -418,6 +461,9 @@ public S3Request createS3Request(S3Parameter s3Parameter) throws GWException { s3Parameter.setOperation(OP_PUT_OBJECT_ACL); return new PutObjectAcl(s3Parameter); } + } else if (s3Parameter.getRequest().getParameter(GWConstants.PARAMETER_LEGAL_HOLD) != null) { + s3Parameter.setOperation(OP_PUT_OBJECT_LEGAL_HOLD); + return new PutObjectLegalHold(s3Parameter); } if (s3Parameter.isAdmin()) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/UploadPart.java b/core/src/com/pspace/ifs/ksan/gw/api/UploadPart.java index b9ef0c9d..4c2d642f 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/UploadPart.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/UploadPart.java @@ -51,10 +51,6 @@ public void process() throws GWException { initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/api/UploadPartCopy.java b/core/src/com/pspace/ifs/ksan/gw/api/UploadPartCopy.java index 33bb8625..fc3c9805 100644 --- a/core/src/com/pspace/ifs/ksan/gw/api/UploadPartCopy.java +++ b/core/src/com/pspace/ifs/ksan/gw/api/UploadPartCopy.java @@ -38,6 +38,7 @@ import com.pspace.ifs.ksan.libs.multipart.Multipart; import com.pspace.ifs.ksan.libs.PrintStack; import com.pspace.ifs.ksan.libs.DiskManager; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.objmanager.Metadata; @@ -59,11 +60,7 @@ public void process() throws GWException { String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); + GWUtils.checkCors(s3Parameter); if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { @@ -97,7 +94,7 @@ public void process() throws GWException { } try { - copySource = URLDecoder.decode(copySource, GWConstants.CHARSET_UTF_8); + copySource = URLDecoder.decode(copySource, Constants.CHARSET_UTF_8); logger.info(GWConstants.LOG_UPLOAD_PART_COPY_SOURCE, copySource); } catch (UnsupportedEncodingException e) { PrintStack.logging(logger, e); @@ -131,6 +128,7 @@ public void process() throws GWException { srcVersionId = source[1].replaceAll(GWConstants.DOUBLE_QUOTE, ""); } + logger.info("source bucket: {}, key: {}, versionId: {},", srcBucket, srcObjectName, srcVersionId); String versioningStatus = getBucketVersioning(srcBucket); Metadata srcMeta = null; @@ -152,7 +150,7 @@ public void process() throws GWException { logger.debug(GWConstants.LOG_SOURCE_INFO, srcBucket, srcObjectName, srcVersionId); - srcMeta.setAcl(GWUtils.makeOriginalXml(srcMeta.getAcl(), s3Parameter)); + // srcMeta.setAcl(GWUtils.makeOriginalXml(srcMeta.getAcl(), s3Parameter)); checkGrantObject(s3Parameter.isPublicAccess(), srcMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); // get metadata @@ -246,7 +244,9 @@ public void process() throws GWException { Metadata objMeta = createLocal(multipart.getDiskPoolId(), bucket, object, "null"); String path = DiskManager.getInstance().getLocalPath(objMeta.getPrimaryDisk().getId()); - + if (path == null) { + path = DiskManager.getInstance().getPath(objMeta.getPrimaryDisk().getId()); + } // Metadata objMeta = createCopy(srcBucket, srcObjectName, srcVersionId, bucket, object); // String path = DiskManager.getInstance().getLocalPath(objMeta.getPrimaryDisk().getId()); @@ -268,7 +268,7 @@ public void process() throws GWException { objMultipart.finishSingleUpload(uploadId, Integer.parseInt(partNumber)); s3Parameter.setFileSize(s3Object.getFileSize()); - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); try (Writer writer = s3Parameter.getResponse().getWriter()) { s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/AzuRequest.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/AzuRequest.java new file mode 100644 index 00000000..a0eb2061 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/AzuRequest.java @@ -0,0 +1,282 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import java.text.SimpleDateFormat; +import java.util.Date; +import java.util.Locale; +import java.util.TimeZone; +import java.util.List; + +import org.slf4j.Logger; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.objmanager.ObjManager; +import com.pspace.ifs.ksan.gw.object.objmanager.ObjManagerHelper; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.objmanager.Bucket; +import com.pspace.ifs.ksan.objmanager.ObjManagerException.ResourceAlreadyExistException; +import com.pspace.ifs.ksan.objmanager.ObjManagerException.ResourceNotFoundException; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.libs.identity.S3BucketSimpleInfo; +import com.pspace.ifs.ksan.libs.identity.ObjectListParameter; +import com.pspace.ifs.ksan.libs.identity.S3ObjectList; + +public abstract class AzuRequest { + protected AzuParameter azuParameter; + protected ObjManager objManager; + protected Bucket container; + protected Logger logger; + + public AzuRequest(AzuParameter parameter) { + this.azuParameter = parameter; + } + + public abstract void process() throws AzuException; + + public void setObjManager() throws Exception { + objManager = ObjManagerHelper.getInstance().getObjManager(); + } + + public void releaseObjManager() throws Exception { + ObjManagerHelper.getInstance().returnObjManager(objManager); + } + + protected boolean isExistContainer(String containerName) throws AzuException { + boolean result = false; + try { + setObjManager(); + result = objManager.isBucketExist(containerName); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + return result; + } + + protected void initContainerInfo(String containerName) throws AzuException { + try { + setObjManager(); + container = objManager.getBucket(containerName); + } catch (ResourceNotFoundException e) { + throw new AzuException(AzuErrorCode.NO_SUCH_CONTAINER, azuParameter); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + + if (container == null) { + logger.info(AzuConstants.LOG_CONTAINER_IS_NOT_EXIST, containerName); + throw new AzuException(AzuErrorCode.NO_SUCH_CONTAINER, azuParameter); + } + } + + protected int createContainer(Bucket container) throws AzuException { + int result = 0; + try { + setObjManager(); + result = objManager.createBucket(container); + } catch (ResourceAlreadyExistException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.CONTAINER_ALREADY_EXISTS, azuParameter); + } catch(ResourceNotFoundException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.INTERNAL_SERVER_ERROR, azuParameter); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.INTERNAL_SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + + if (result != 0) { + throw new AzuException(AzuErrorCode.INTERNAL_SERVER_DB_ERROR, azuParameter); + } + + return result; + } + + protected void deleteContainer(String container) throws AzuException { + boolean result = false; + try { + setObjManager(); + objManager.removeBucket(container); + // result = objManager.isBucketDelete(container); + // if (result) { + // objManager.removeBucket(container); + // } + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + + // if (!result) { + // logger.info(GWConstants.LOG_REQUEST_BUCKET_IS_NOT_EMPTY); + // throw new AzuException(AzuErrorCode.BUCKET_NOT_EMPTY, azuParameter); + // } + } + + protected List listContainerSimpleInfo(String userName, String userId) throws AzuException { + List bucketList = null; + try { + setObjManager(); + bucketList = objManager.listBucketSimpleInfo(userName, userId); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + + return bucketList; + } + + protected Metadata open(String containerName, String blobName) throws AzuException { + Metadata meta = null; + try { + setObjManager(); + meta = objManager.open(containerName, blobName, "null"); + } catch (ResourceNotFoundException e) { + throw new AzuException(AzuErrorCode.NO_SUCH_KEY, azuParameter); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + + return meta; + } + + protected Metadata createLocal(String diskpoolId, String containerName, String blobName, String versionId) throws AzuException { + Metadata meta = null; + try { + setObjManager(); + meta = objManager.createLocal(diskpoolId, containerName, blobName, versionId); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + + return meta; + } + + protected int insertObject(String containerName, String blobName, Metadata data) throws AzuException { + int result = 0; + try { + setObjManager(); + result = objManager.close(containerName, blobName, data); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + + return result; + } + + protected ObjectListParameter listObject(String bucket, S3ObjectList s3ObjectList) throws AzuException { + ObjectListParameter objectListParameter = null; + try { + setObjManager(); + objectListParameter = objManager.listObject(bucket, s3ObjectList); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + + return objectListParameter; + } + + protected void remove(String bucket, String object) throws AzuException { + try { + setObjManager(); + objManager.remove(bucket, object); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + } + + protected String formatDate(Date date) { + SimpleDateFormat formatter = new SimpleDateFormat(AzuConstants.TIME_GMT_FORMAT, new Locale(AzuConstants.LOCALE_EN)); + formatter.setTimeZone(TimeZone.getTimeZone(AzuConstants.TIMEZONE_GMT)); + + return formatter.format(date); + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/AzuRequestFactory.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/AzuRequestFactory.java new file mode 100644 index 00000000..2c5985be --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/AzuRequestFactory.java @@ -0,0 +1,83 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.exception.*; + +public class AzuRequestFactory { + private final Logger logger; + + public AzuRequestFactory() { + logger = LoggerFactory.getLogger(AzuRequestFactory.class); + } + + public AzuRequest createRequest(AzuParameter parameter) throws AzuException { + logger.info("method: {}, category : {}, comp : {}, restype : {}", parameter.getMethod(), parameter.getPathCategory(), parameter.getComp(), parameter.getRestype()); + switch (parameter.getMethod()) { + case AzuConstants.HTTP_METHOD_DELETE: + if (parameter.getRestype().equals(AzuConstants.RESTYPE_CONTAINER)) { + return new DeleteContainer(parameter); + } else if (parameter.getPathCategory().equals(AzuConstants.PATH_CATEGORY_BLOB)) { + return new DeleteBlob(parameter); + } + break; + + case AzuConstants.HTTP_METHOD_GET: + if (parameter.getRestype().equals(AzuConstants.RESTYPE_SERVICE) && parameter.getComp().equals(AzuConstants.COMP_PROPERTIES)) { + return new GetProperties(parameter); + } else if (parameter.getRestype().equals(AzuConstants.RESTYPE_ACCOUNT) && parameter.getComp().equals(AzuConstants.COMP_PROPERTIES)) { + return new SharedKeyAuth(parameter); + } else if (parameter.getPathCategory().equals(AzuConstants.PATH_CATEGORY_ROOT)) { + return new ListContainers(parameter); + } else if (parameter.getPathCategory().equals(AzuConstants.PATH_CATEGORY_CONTAINER)) { + return new ListBlobs(parameter); + } else if (parameter.getPathCategory().equals(AzuConstants.PATH_CATEGORY_BLOB)) { + return new GetBlob(parameter); + } + break; + + case AzuConstants.HTTP_METHOD_HEAD: + if (parameter.getPathCategory().equals(AzuConstants.PATH_CATEGORY_CONTAINER)) { + return new HeadContainer(parameter); + } else if (parameter.getPathCategory().equals(AzuConstants.PATH_CATEGORY_BLOB)) { + return new HeadBlob(parameter); + } + break; + + case AzuConstants.HTTP_METHOD_PUT: + if (parameter.getPathCategory().equals(AzuConstants.PATH_CATEGORY_CONTAINER)) { + return new CreateContainer(parameter); + } else if (parameter.getPathCategory().equals(AzuConstants.PATH_CATEGORY_BLOB)) { + if (parameter.getComp().equals(AzuConstants.COMP_BLOCK)) { + return new PutBlock(parameter); + } else if (parameter.getComp().equals(AzuConstants.COMP_BLOCKLIST)) { + return new PutBlockList(parameter); + } else { + return new PutBlob(parameter); + } + } + break; + + default: + break; + } + + logger.error("Unknown method: {}", parameter.getMethod()); + throw new AzuException(AzuErrorCode.NOT_IMPLEMENTED, parameter); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/CreateContainer.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/CreateContainer.java new file mode 100644 index 00000000..86cc16d9 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/CreateContainer.java @@ -0,0 +1,70 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConfig; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; + +import com.pspace.ifs.ksan.objmanager.Bucket; + +import java.io.BufferedReader; +import java.io.BufferedWriter; +import java.io.File; +import java.io.FileReader; +import java.io.FileWriter; +import java.io.IOException; + +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletResponse; + +public class CreateContainer extends AzuRequest { + + public CreateContainer(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(CreateContainer.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_CREATE_CONTAINER_START); + String containerName = azuParameter.getContainerName(); + + logger.info(AzuConstants.LOG_CREATE_CONTAINER_NAME, containerName); + if (isExistContainer(containerName)) { + throw new AzuException(AzuErrorCode.CONTAINER_ALREADY_EXISTS); + } + + int result = 0; + String diskpoolId = azuParameter.getUser().getUserDefaultDiskpoolId(); + logger.info("user default diskpoolId : {}", diskpoolId); + + Bucket bucket = new Bucket(); + bucket.setName(containerName); + bucket.setUserId(azuParameter.getUser().getUserId()); + bucket.setUserName(azuParameter.getUser().getUserName()); + bucket.setAcl(AzuConstants.EMPTY_STRING); + bucket.setDiskPoolId(diskpoolId); + + result = createContainer(bucket); + if (result != 0) { + throw new AzuException(AzuErrorCode.INTERNAL_SERVER_DB_ERROR, azuParameter); + } + + azuParameter.getResponse().setStatus(HttpServletResponse.SC_CREATED); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/DeleteBlob.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/DeleteBlob.java new file mode 100644 index 00000000..96fb388a --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/DeleteBlob.java @@ -0,0 +1,60 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.gw.object.S3Object; +import com.pspace.ifs.ksan.gw.object.AzuObjectOperation; + +import java.io.File; + +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletResponse; + +public class DeleteBlob extends AzuRequest { + + public DeleteBlob(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(DeleteBlob.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_DELETE_BLOB_START); + + String containerName = azuParameter.getContainerName(); + String blobName = azuParameter.getBlobName(); + String versionId = GWConstants.VERSIONING_DISABLE_TAIL; + Metadata objMeta = null; + try { + objMeta = open(containerName, blobName); + remove(azuParameter.getContainerName(), azuParameter.getBlobName()); + AzuObjectOperation azuObjectOperation = new AzuObjectOperation(objMeta, null, azuParameter, versionId); + azuObjectOperation.deleteObject(); + } catch (AzuException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.NO_SUCH_KEY, azuParameter); + } + + azuParameter.getResponse().setStatus(HttpServletResponse.SC_ACCEPTED); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/DeleteContainer.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/DeleteContainer.java new file mode 100644 index 00000000..df170bca --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/DeleteContainer.java @@ -0,0 +1,48 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; + +import java.io.File; + +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletResponse; + +public class DeleteContainer extends AzuRequest { + + public DeleteContainer(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(DeleteContainer.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_DELETE_CONTAINER_START); + + String containerName = azuParameter.getContainerName(); + + if (isExistContainer(containerName)) { + deleteContainer(containerName); + azuParameter.getResponse().setStatus(HttpServletResponse.SC_ACCEPTED); + } else { + azuParameter.getResponse().setStatus(HttpServletResponse.SC_NOT_FOUND); + } + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/GetBlob.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/GetBlob.java new file mode 100644 index 00000000..48048439 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/GetBlob.java @@ -0,0 +1,93 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.gw.object.S3Object; +import com.pspace.ifs.ksan.gw.object.AzuObjectOperation; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; + +import java.io.BufferedReader; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileReader; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.attribute.BasicFileAttributes; +import java.util.Date; + +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletResponse; + +public class GetBlob extends AzuRequest { + + public GetBlob(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(GetBlob.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_GET_BLOB_START); + + String containerName = azuParameter.getContainerName(); + String blobName = azuParameter.getBlobName(); + String versionId = GWConstants.VERSIONING_DISABLE_TAIL; + + Metadata objMeta = open(containerName, blobName); + S3Metadata s3Metadata = new S3Metadata(); + ObjectMapper jsonMapper = new ObjectMapper(); + try { + logger.debug(GWConstants.LOG_META, objMeta.getMeta()); + s3Metadata = jsonMapper.readValue(objMeta.getMeta(), S3Metadata.class); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + String range = azuParameter.getRequest().getHeader(AzuConstants.X_MS_RANGE); + logger.debug("range : {}", range); + + AzuObjectOperation azuObjectOperation = new AzuObjectOperation(objMeta, null, azuParameter, versionId); + try { + azuObjectOperation.getObject(range); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + azuParameter.getResponse().setCharacterEncoding(AzuConstants.CHARSET_UTF_8); + azuParameter.getResponse().setContentLength(s3Metadata.getContentLength().intValue()); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_LASTMODIFIED, formatDate(s3Metadata.getLastModified())); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_X_MS_CREATION_TIME, formatDate(s3Metadata.getCreationDate())); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_X_MS_BLOB_TYPE, AzuConstants.BLOB_TYPE_BLOCKBLOB); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_X_MS_LEASE_STATE, AzuConstants.LEASE_STATE_AVAILABLE); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_X_MS_LEASE_STATUS, AzuConstants.LEASE_STATUS_UNLOCKED); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_ETAG, AzuConstants.ETAG_DEFAULT); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_CONTENT_MD5, s3Metadata.getContentMD5()); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/GetProperties.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/GetProperties.java new file mode 100644 index 00000000..027ed071 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/GetProperties.java @@ -0,0 +1,128 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.libs.PrintStack; + +import java.io.File; +import java.io.IOException; +import java.io.Writer; +import java.util.Date; + +import javax.xml.stream.XMLOutputFactory; +import javax.xml.stream.XMLStreamException; +import javax.xml.stream.XMLStreamWriter; + +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletResponse; + +public class GetProperties extends AzuRequest { + + public GetProperties(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(GetProperties.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_GET_PROPERTIES_START); + + azuParameter.getResponse().setCharacterEncoding(AzuConstants.CHARSET_UTF_8); + try { + XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); + try (Writer writer = azuParameter.getResponse().getWriter()) { + azuParameter.getResponse().setContentType(AzuConstants.CONTENT_TYPE_XML); + XMLStreamWriter xmlStreamWriter = xmlOutputFactory.createXMLStreamWriter(writer); + xmlStreamWriter.writeStartDocument(); + xmlStreamWriter.writeStartElement(AzuConstants.STORAGE_SERVICE_PROPERTIES); + + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_LOGGING); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_VERSION); + xmlStreamWriter.writeCharacters(AzuConstants.VERSION_VALUE); + xmlStreamWriter.writeEndElement(); // End Version + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_DELETE); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_TRUE); + xmlStreamWriter.writeEndElement(); // End Delete + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_READ); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_TRUE); + xmlStreamWriter.writeEndElement(); // End Read + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_WRITE); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_TRUE); + xmlStreamWriter.writeEndElement(); // End Write + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_RETENTION_POLICY); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ENABLED); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_FALSE); + xmlStreamWriter.writeEndElement(); // End Enabled + xmlStreamWriter.writeEndElement(); // End RetentionPolicy + xmlStreamWriter.writeEndElement(); // End Logging + + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_HOUR_METRICS); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_VERSION); + xmlStreamWriter.writeCharacters(AzuConstants.VERSION_VALUE); + xmlStreamWriter.writeEndElement(); // End Version + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ENABLED); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_FALSE); + xmlStreamWriter.writeEndElement(); // End Enabled + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_RETENTION_POLICY); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ENABLED); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_FALSE); + xmlStreamWriter.writeEndElement(); // End Enabled + xmlStreamWriter.writeEndElement(); // End RetentionPolicy + xmlStreamWriter.writeEndElement(); // End HourMetrics + + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_MINUTE_METRICS); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_VERSION); + xmlStreamWriter.writeCharacters(AzuConstants.VERSION_VALUE); + xmlStreamWriter.writeEndElement(); // End Version + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ENABLED); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_FALSE); + xmlStreamWriter.writeEndElement(); // End Enabled + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_RETENTION_POLICY); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ENABLED); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_FALSE); + xmlStreamWriter.writeEndElement(); // End Enabled + xmlStreamWriter.writeEndElement(); // End RetentionPolicy + xmlStreamWriter.writeEndElement(); // End MinuteMetrics + + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CORS); + xmlStreamWriter.writeEndElement(); // End Enabled + + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_DEFAULT_SERVICE_VERSION); + xmlStreamWriter.writeCharacters(AzuConstants.SERVICE_VERSION_VALUE); + xmlStreamWriter.writeEndElement(); // End DefaultServiceVersion + + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_STATIC_WEBSITE); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ENABLED); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_FALSE); + xmlStreamWriter.writeEndElement(); // End Enabled + xmlStreamWriter.writeEndElement(); // End StaticWebsite + + xmlStreamWriter.writeEndElement(); // End StorageServiceProperties + xmlStreamWriter.flush(); + } catch (XMLStreamException | IOException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } catch (Exception e) { + PrintStack.logging(logger, e); + } + + azuParameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/HeadBlob.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/HeadBlob.java new file mode 100644 index 00000000..f376af1c --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/HeadBlob.java @@ -0,0 +1,78 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; + +import java.io.BufferedReader; +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.attribute.BasicFileAttributes; +import java.util.Date; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletResponse; + +public class HeadBlob extends AzuRequest { + + public HeadBlob(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(HeadBlob.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_HEAD_BLOB_START); + + Metadata objMeta = open(azuParameter.getContainerName(), azuParameter.getBlobName()); + ObjectMapper objectMapper = new ObjectMapper(); + S3Metadata s3Metadata = null; + try { + s3Metadata = objectMapper.readValue(objMeta.getMeta(), S3Metadata.class); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + azuParameter.getResponse().setCharacterEncoding(AzuConstants.CHARSET_UTF_8); + try { + azuParameter.getResponse().setStatus(HttpServletResponse.SC_OK); + azuParameter.getResponse().setContentLength(s3Metadata.getContentLength().intValue()); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_LASTMODIFIED, formatDate(s3Metadata.getLastModified())); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_X_MS_CREATION_TIME, formatDate(s3Metadata.getCreationDate())); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_X_MS_BLOB_TYPE, AzuConstants.BLOB_TYPE_BLOCKBLOB); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_X_MS_LEASE_STATE, AzuConstants.LEASE_STATE_AVAILABLE); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_X_MS_LEASE_STATUS, AzuConstants.LEASE_STATUS_UNLOCKED); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_ETAG, AzuConstants.ETAG_DEFAULT); + azuParameter.getResponse().addHeader(AzuConstants.HEADER_CONTENT_MD5, s3Metadata.getContentMD5()); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/HeadContainer.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/HeadContainer.java new file mode 100644 index 00000000..31fd5d46 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/HeadContainer.java @@ -0,0 +1,40 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; + +import java.io.File; + +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletResponse; + +public class HeadContainer extends AzuRequest { + + public HeadContainer(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(HeadContainer.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_HAED_CONTAINER_START); + initContainerInfo(azuParameter.getContainerName()); + + azuParameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/ListBlobs.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/ListBlobs.java new file mode 100644 index 00000000..71a037aa --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/ListBlobs.java @@ -0,0 +1,206 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.data.azure.DataListBlobs; +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.libs.identity.ObjectListParameter; +import com.pspace.ifs.ksan.libs.identity.S3ObjectList; + +import java.util.Map.Entry; +import java.io.BufferedReader; +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.io.Writer; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.attribute.BasicFileAttributes; +import java.util.Date; + +import javax.xml.stream.XMLOutputFactory; +import javax.xml.stream.XMLStreamException; +import javax.xml.stream.XMLStreamWriter; + +import jakarta.servlet.http.HttpServletResponse; + +import org.slf4j.LoggerFactory; + +import com.google.common.base.Strings; + +public class ListBlobs extends AzuRequest { + + public ListBlobs(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(ListBlobs.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_LIST_BLOB_START); + + String containerName = azuParameter.getContainerName(); + String prefix = azuParameter.getPrefix(); + String maxResults = azuParameter.getMaxResults(); + String delimiter = azuParameter.getDelimiter(); + String marker = azuParameter.getMarker(); + String prefixPath = null; + + S3ObjectList s3ObjectList = new S3ObjectList(); + if (!Strings.isNullOrEmpty(maxResults)) { + try { + if (Integer.valueOf(maxResults) <= 0) { + throw new AzuException(AzuErrorCode.BAD_REQUEST, azuParameter); + } else if (Integer.valueOf(maxResults) > 1000) { + s3ObjectList.setMaxKeys(GWConstants.DEFAULT_MAX_KEYS); + } else { + s3ObjectList.setMaxKeys(maxResults); + } + } catch (NumberFormatException e) { + throw new AzuException(AzuErrorCode.BAD_REQUEST, azuParameter); + } + } else { + s3ObjectList.setMaxKeys(GWConstants.DEFAULT_MAX_KEYS); + } + + s3ObjectList.setDelimiter(delimiter); + s3ObjectList.setEncodingType(AzuConstants.CHARSET_UTF_8); + s3ObjectList.setMarker(marker); + s3ObjectList.setPrefix(prefix); + logger.debug("container : {}", containerName); + logger.debug("maxResults : {}", s3ObjectList.getMaxKeys()); + logger.debug("delimiter : {}", s3ObjectList.getDelimiter()); + logger.debug("marker : {}", s3ObjectList.getMarker()); + logger.debug("prefix : {}", s3ObjectList.getPrefix()); + + ObjectListParameter objectListParameter = listObject(containerName, s3ObjectList); + + azuParameter.getResponse().setCharacterEncoding(AzuConstants.CHARSET_UTF_8); + XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); + try (Writer writer = azuParameter.getResponse().getWriter()) { + azuParameter.getResponse().setContentType(AzuConstants.CONTENT_TYPE_XML); + XMLStreamWriter xmlStreamWriter = xmlOutputFactory.createXMLStreamWriter(writer); + xmlStreamWriter.writeStartDocument(); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ENUMERATION_RESULT); + xmlStreamWriter.writeAttribute(AzuConstants.STORAGE_SERVICE_PROPERTIES, AzuConstants.XML_ATTRIBUTE_SERVICE_ENDPOINT_VALUE + azuParameter.getUserName()); + xmlStreamWriter.writeAttribute(AzuConstants.XML_ATTRIBUTE_CONTAINER_NAME, azuParameter.getContainerName()); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_PREFIX); + if (!Strings.isNullOrEmpty(prefix)) { + xmlStreamWriter.writeCharacters(prefix); + logger.debug("prefix : {}", prefix); + } + xmlStreamWriter.writeEndElement(); // End Prefix + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_MARKER); + if (!Strings.isNullOrEmpty(marker)) { + xmlStreamWriter.writeCharacters(marker); + } + xmlStreamWriter.writeEndElement(); // End Marker + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_MAX_RESULT); + xmlStreamWriter.writeCharacters(maxResults); + xmlStreamWriter.writeEndElement(); // End MaxResults + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_DELIMITER); + if (!Strings.isNullOrEmpty(delimiter)) { + xmlStreamWriter.writeCharacters(delimiter); + } + xmlStreamWriter.writeEndElement(); // End Delimiter + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_BLOBS); + + for (S3Metadata s3Metadata : objectListParameter.getObjects()) { + String blobName = s3Metadata.getName(); + logger.info("bolb : {}", blobName); + + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_BLOB); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_NAME); + xmlStreamWriter.writeCharacters(blobName); + xmlStreamWriter.writeEndElement(); // end Name + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_PROPERTIES); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CREATION_TIME); + xmlStreamWriter.writeCharacters(formatDate(s3Metadata.getCreationDate())); + xmlStreamWriter.writeEndElement(); // End Creation-Time + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_LAST_MODIFIED); + xmlStreamWriter.writeCharacters(formatDate(s3Metadata.getLastModified())); + xmlStreamWriter.writeEndElement(); // End Last-Modified + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ETAG); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_ETAG); + xmlStreamWriter.writeEndElement(); // End Etag + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CONTENT_LENGTH); + xmlStreamWriter.writeCharacters(Long.toString(s3Metadata.getContentLength())); + xmlStreamWriter.writeEndElement(); // End Content-Length + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CONTENT_TYPE); + xmlStreamWriter.writeCharacters(s3Metadata.getContentType()); + xmlStreamWriter.writeEndElement(); // End Content-Type + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CONTENT_ENCODING); + xmlStreamWriter.writeEndElement(); // End Content-Encoding + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CONTENT_LANGUAGE); + xmlStreamWriter.writeEndElement(); // End Content-Language + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CONTENT_MD5); + xmlStreamWriter.writeCharacters(s3Metadata.getETag()); + xmlStreamWriter.writeEndElement(); // End Content-MD5 + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CONTENT_DISPOSITION); + xmlStreamWriter.writeEndElement(); // End Content-Disposition + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CACHE_CONTROL); + xmlStreamWriter.writeEndElement(); // End Cache-Control/ + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_BLOB_TYPE); + xmlStreamWriter.writeCharacters(AzuConstants.BLOB_TYPE_BLOCKBLOB); + xmlStreamWriter.writeEndElement(); // End BlobType + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_LEASE_STATUS); + xmlStreamWriter.writeCharacters(AzuConstants.LEASE_STATUS_UNLOCKED); + xmlStreamWriter.writeEndElement(); // End LeaseStatus + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_LEASE_STATE); + xmlStreamWriter.writeCharacters(AzuConstants.LEASE_STATE_AVAILABLE); + xmlStreamWriter.writeEndElement(); // End LeaseState + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_SERVER_ENCRYPTED); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_TRUE); + xmlStreamWriter.writeEndElement(); // End ServerEncrypted + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ACCESS_TIER); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_HOT); + xmlStreamWriter.writeEndElement(); // End AccessTier + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ACCESS_TIER_INFERRED); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_TRUE); + xmlStreamWriter.writeEndElement(); // End AccessTierInferred + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ACCESS_TIER_CHANGE_TIME); + xmlStreamWriter.writeCharacters(formatDate(s3Metadata.getLastModified())); + xmlStreamWriter.writeEndElement(); // End AccessTierChangeTime + xmlStreamWriter.writeEndElement(); // End Properties + xmlStreamWriter.writeEndElement(); // End Blob + } + + for (Entry entry : objectListParameter.getCommonPrefixes().entrySet()) { + logger.info("dir : {}", entry.getValue()); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_BLOB_PREFIX); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_NAME); + xmlStreamWriter.writeCharacters(entry.getValue()); + xmlStreamWriter.writeEndElement(); // end Name + xmlStreamWriter.writeEndElement(); // end BlobPrefix + } + + xmlStreamWriter.writeEndElement(); // End Blobs + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_NEXT_MARKER); + xmlStreamWriter.writeEndElement(); // End NextMarker + xmlStreamWriter.writeEndElement(); // End EnumerationResults + xmlStreamWriter.flush(); + } catch (XMLStreamException | IOException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + azuParameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/ListContainers.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/ListContainers.java new file mode 100644 index 00000000..a9ca0767 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/ListContainers.java @@ -0,0 +1,111 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.identity.S3BucketSimpleInfo; + +import java.io.File; +import java.io.IOException; +import java.io.Writer; +import java.util.Date; +import java.util.List; + +import javax.xml.stream.XMLOutputFactory; +import javax.xml.stream.XMLStreamException; +import javax.xml.stream.XMLStreamWriter; + +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletResponse; + +public class ListContainers extends AzuRequest { + + public ListContainers(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(ListContainers.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_LIST_CONTAINER_START); + + List containerList = listContainerSimpleInfo(azuParameter.getUser().getUserName(), azuParameter.getUser().getUserId()); + + azuParameter.getResponse().setCharacterEncoding(AzuConstants.CHARSET_UTF_8); + try { + XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); + try (Writer writer = azuParameter.getResponse().getWriter()) { + azuParameter.getResponse().setContentType(AzuConstants.CONTENT_TYPE_XML); + XMLStreamWriter xmlStreamWriter = xmlOutputFactory.createXMLStreamWriter(writer); + xmlStreamWriter.writeStartDocument(); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ENUMERATION_RESULT); + xmlStreamWriter.writeAttribute(AzuConstants.STORAGE_SERVICE_PROPERTIES, AzuConstants.XML_ATTRIBUTE_SERVICE_ENDPOINT_VALUE + azuParameter.getUserName()); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_PREFIX); + xmlStreamWriter.writeEndElement(); // End Prefix + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_MAX_RESULT); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_LISTCONTAINER_MAX_RESULT_VALUE); + xmlStreamWriter.writeEndElement(); // End MaxResults + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CONTAINERS); + + for (S3BucketSimpleInfo container : containerList) { + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CONTAINER); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_NAME); + xmlStreamWriter.writeCharacters(container.getBucketName()); + xmlStreamWriter.writeEndElement(); // end Name + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_PROPERTIES); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_LAST_MODIFIED); + xmlStreamWriter.writeCharacters(formatDate(container.getCreateDate())); + xmlStreamWriter.writeEndElement(); // End Last-Modified + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ETAG); + xmlStreamWriter.writeCharacters(AzuConstants.ETAG_DEFAULT); + xmlStreamWriter.writeEndElement(); // End Etag + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_LEASE_STATUS); + xmlStreamWriter.writeCharacters(AzuConstants.LEASE_STATUS_UNLOCKED); + xmlStreamWriter.writeEndElement(); // End LeaseStatus + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_LEASE_STATE); + xmlStreamWriter.writeCharacters(AzuConstants.LEASE_STATE_AVAILABLE); + xmlStreamWriter.writeEndElement(); // End LeaseState + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_HAS_IMMUTABILITY_POLICY); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_FALSE); + xmlStreamWriter.writeEndElement(); // End HasImmutabilityPolicy + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_HAS_LEGALHOLD); + xmlStreamWriter.writeCharacters(AzuConstants.XML_ELEMENT_VALUE_FALSE); + xmlStreamWriter.writeEndElement(); // End HasLegalHold + xmlStreamWriter.writeEndElement(); // End Properties + xmlStreamWriter.writeEndElement(); // End Container + + logger.debug(AzuConstants.LOG_LIST_CONTAINER_NAME_DATE, container.getBucketName(), formatDate(container.getCreateDate())); + } + + xmlStreamWriter.writeEndElement(); // End Containers + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_NEXT_MARKER); + xmlStreamWriter.writeEndElement(); // End NextMarker + xmlStreamWriter.writeEndElement(); // End EnumerationResults + xmlStreamWriter.flush(); + } catch (XMLStreamException | IOException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } catch (Exception e) { + PrintStack.logging(logger, e); + } + + azuParameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/PutBlob.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/PutBlob.java new file mode 100644 index 00000000..f3e2fba9 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/PutBlob.java @@ -0,0 +1,154 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import static com.google.common.io.BaseEncoding.base64; + +import com.google.common.base.Strings; + +import com.fasterxml.jackson.annotation.JsonInclude.Include; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; + +import com.pspace.ifs.ksan.gw.data.azure.DataPutBlob; +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConfig; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.gw.object.S3Object; +import com.pspace.ifs.ksan.gw.object.AzuObjectOperation; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.InputStream; +import java.io.StringReader; +import java.security.MessageDigest; +import java.util.Date; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; + +import org.slf4j.LoggerFactory; +import org.w3c.dom.Document; +import org.w3c.dom.Element; +import org.w3c.dom.Node; +import org.w3c.dom.NodeList; +import org.xml.sax.InputSource; + +import jakarta.servlet.http.HttpServletResponse; + +public class PutBlob extends AzuRequest { + + public PutBlob(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(PutBlob.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_CREATE_BLOB_START); + + String containerName = azuParameter.getContainerName(); + String blobName = azuParameter.getBlobName(); + String storageClass = GWConstants.AWS_TIER_STANTARD; + String diskpoolId = ""; + try { + diskpoolId = azuParameter.getUser().getUserDefaultDiskpoolId(); + } catch (Exception e) { + PrintStack.logging(logger, e); + } + + DataPutBlob dataCreateBlob = new DataPutBlob(azuParameter); + dataCreateBlob.extract(); + + String contentsLength = dataCreateBlob.getContentLength(); + long blobLength = Long.parseLong(contentsLength); + String contentMD5 = dataCreateBlob.getContentMD5(); + + logger.debug("contentsLength : {}, {}", contentsLength, blobLength); + + String versionId = GWConstants.VERSIONING_DISABLE_TAIL; + Metadata objMeta = null; + boolean isExist = false; + try { + // check exist object + objMeta = open(containerName, blobName); + isExist = true; + } catch (AzuException e) { + logger.info(e.getMessage()); + // reset error code + azuParameter.setErrorCode(GWConstants.EMPTY_STRING); + objMeta = createLocal(diskpoolId, containerName, blobName, versionId); + } + + S3Metadata s3Metadata = new S3Metadata(); + ObjectMapper jsonMapper = new ObjectMapper(); + if (isExist) { + try { + logger.debug(GWConstants.LOG_META, objMeta.getMeta()); + s3Metadata = jsonMapper.readValue(objMeta.getMeta(), S3Metadata.class); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } else { + s3Metadata.setCreationDate(new Date()); + } + + s3Metadata.setContentType(dataCreateBlob.getContentType()); + s3Metadata.setOwnerId(azuParameter.getUser().getUserId()); + s3Metadata.setOwnerName(azuParameter.getUser().getUserName()); + s3Metadata.setContentLength(blobLength); + + AzuObjectOperation azuObjectOperation = new AzuObjectOperation(objMeta, s3Metadata, azuParameter, versionId); + S3Object s3Object = azuObjectOperation.putObject(); + + s3Metadata.setETag(s3Object.getEtag()); + s3Metadata.setContentLength(s3Object.getFileSize()); + s3Metadata.setTier(storageClass); + s3Metadata.setLastModified(s3Object.getLastModified()); + s3Metadata.setDeleteMarker(s3Object.getDeleteMarker()); + s3Metadata.setVersionId(versionId); + + logger.info("MD5 check, receive : {}, result : {}", contentMD5, s3Object.getEtag()); + + String jsonmeta = ""; + try { + jsonmeta = jsonMapper.writeValueAsString(s3Metadata); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + logger.debug(AzuConstants.LOG_CREATE_BLOB_PRIMARY_DISK_ID, objMeta.getPrimaryDisk().getId()); + try { + objMeta.set(s3Object.getEtag(), AzuConstants.EMPTY_STRING, jsonmeta, AzuConstants.EMPTY_STRING, s3Object.getFileSize()); + objMeta.setVersionId(versionId, GWConstants.OBJECT_TYPE_FILE, true); + int result = insertObject(containerName, blobName, objMeta); + } catch (AzuException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + azuParameter.getResponse().setStatus(HttpServletResponse.SC_CREATED); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/PutBlock.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/PutBlock.java new file mode 100644 index 00000000..541efa00 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/PutBlock.java @@ -0,0 +1,117 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import static com.google.common.io.BaseEncoding.base64; + +import com.google.common.base.Strings; + +import com.fasterxml.jackson.annotation.JsonInclude.Include; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; + +import com.pspace.ifs.ksan.gw.data.azure.DataPutBlock; +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConfig; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.gw.object.S3Object; +import com.pspace.ifs.ksan.gw.object.AzuObjectOperation; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.InputStream; +import java.io.StringReader; +import java.security.MessageDigest; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; + +import org.slf4j.LoggerFactory; +import org.w3c.dom.Document; +import org.w3c.dom.Element; +import org.w3c.dom.Node; +import org.w3c.dom.NodeList; +import org.xml.sax.InputSource; + +import jakarta.servlet.http.HttpServletResponse; + +public class PutBlock extends AzuRequest { + + public PutBlock(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(PutBlock.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_UPLOAD_BLOCK_START); + + String containerName = azuParameter.getContainerName(); + String blobName = azuParameter.getBlobName(); + String storageClass = GWConstants.AWS_TIER_STANTARD; + String diskpoolId = ""; + try { + diskpoolId = azuParameter.getUser().getUserDefaultDiskpoolId(); + } catch (Exception e) { + PrintStack.logging(logger, e); + } + + DataPutBlock dataUploadBlock = new DataPutBlock(azuParameter); + dataUploadBlock.extract(); + String blockId = dataUploadBlock.getBlockId(); + if (Strings.isNullOrEmpty(blockId)) { + throw new AzuException(AzuErrorCode.BAD_REQUEST, azuParameter); + } + + String contentsLength = dataUploadBlock.getContentLength(); + long blockLength = Long.parseLong(contentsLength); + + String versionId = GWConstants.VERSIONING_DISABLE_TAIL; + Metadata objMeta = null; + try { + // check exist object + objMeta = open(containerName, blobName); + } catch (AzuException e) { + logger.info(e.getMessage()); + // reset error code + azuParameter.setErrorCode(GWConstants.EMPTY_STRING); + objMeta = createLocal(diskpoolId, containerName, blobName, versionId); + objMeta.set(AzuConstants.EMPTY_STRING, AzuConstants.EMPTY_STRING, AzuConstants.EMPTY_STRING, AzuConstants.EMPTY_STRING, 0); + objMeta.setVersionId(versionId, GWConstants.OBJECT_TYPE_FILE, true); + try { + insertObject(containerName, blobName, objMeta); + } catch (AzuException e1) { + // duplicate key error + logger.info(e1.getMessage()); + objMeta = open(containerName, blobName); + } + } + + AzuObjectOperation azuObjectOperation = new AzuObjectOperation(objMeta, null, azuParameter, versionId); + S3Object s3Object = azuObjectOperation.uploadBlock(blockId, blockLength); + logger.info("blockId : {}, etag : {}", blockId, s3Object.getEtag()); + + azuParameter.getResponse().setStatus(HttpServletResponse.SC_CREATED); + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/PutBlockList.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/PutBlockList.java new file mode 100644 index 00000000..bca551cd --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/PutBlockList.java @@ -0,0 +1,136 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.data.azure.DataPutBlockList; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.format.BlockList; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; + +import static com.google.common.io.BaseEncoding.base64; +import com.google.common.base.Strings; + +import com.fasterxml.jackson.annotation.JsonInclude.Include; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; + +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.utils.AzuConfig; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.AzuUtils; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.gw.object.S3Object; +import com.pspace.ifs.ksan.gw.object.AzuObjectOperation; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.InputStream; +import java.io.StringReader; +import java.security.MessageDigest; +import java.util.Date; + +import javax.xml.parsers.DocumentBuilder; +import javax.xml.parsers.DocumentBuilderFactory; + +import org.slf4j.LoggerFactory; +import org.w3c.dom.Document; +import org.w3c.dom.Element; +import org.w3c.dom.Node; +import org.w3c.dom.NodeList; +import org.xml.sax.InputSource; + +import jakarta.servlet.http.HttpServletResponse; + +public class PutBlockList extends AzuRequest { + + public PutBlockList(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(PutBlockList.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_COMPLETE_BLOCK_LIST_START); + String containerName = azuParameter.getContainerName(); + String blobName = azuParameter.getBlobName(); + + DataPutBlockList dataCompleteBlockList = new DataPutBlockList(azuParameter); + dataCompleteBlockList.extract(); + String blockListXml = dataCompleteBlockList.getXml(); + logger.info("blocklist xml : {}", blockListXml); + + XmlMapper xmlMapper = new XmlMapper(); + BlockList blockList = new BlockList(); + try { + blockList = xmlMapper.readValue(blockListXml, BlockList.class); + } catch (JsonMappingException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.INTERNAL_SERVER_ERROR, azuParameter); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.INTERNAL_SERVER_ERROR, azuParameter); + } + + for (String blockId : blockList.latests) { + logger.info("-- blockId : {}", blockId); + } + + String versionId = GWConstants.VERSIONING_DISABLE_TAIL; + Metadata objMeta = open(containerName, blobName); + + S3Metadata s3Metadata = new S3Metadata(); + s3Metadata.setCreationDate(new Date()); + s3Metadata.setContentType(dataCompleteBlockList.getBlobContentType()); + s3Metadata.setOwnerId(azuParameter.getUser().getUserId()); + s3Metadata.setOwnerName(azuParameter.getUser().getUserName()); + + AzuObjectOperation azuObjectOperation = new AzuObjectOperation(objMeta, s3Metadata, azuParameter, versionId); + S3Object s3Object = azuObjectOperation.completeBlockList(blockList.latests); + + s3Metadata.setETag(s3Object.getEtag()); + s3Metadata.setContentLength(s3Object.getFileSize()); + s3Metadata.setTier(GWConstants.AWS_TIER_STANTARD); + s3Metadata.setLastModified(s3Object.getLastModified()); + s3Metadata.setDeleteMarker(s3Object.getDeleteMarker()); + s3Metadata.setVersionId(versionId); + + logger.info("MD5 check, receive : {}, result : {}", dataCompleteBlockList.getBlobContentMD5(), s3Object.getEtag()); + + ObjectMapper jsonMapper = new ObjectMapper(); + String jsonmeta = ""; + try { + jsonmeta = jsonMapper.writeValueAsString(s3Metadata); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + logger.debug(AzuConstants.LOG_CREATE_BLOB_PRIMARY_DISK_ID, objMeta.getPrimaryDisk().getId()); + try { + objMeta.set(s3Object.getEtag(), AzuConstants.EMPTY_STRING, jsonmeta, AzuConstants.EMPTY_STRING, s3Object.getFileSize()); + objMeta.setVersionId(versionId, GWConstants.OBJECT_TYPE_FILE, true); + int result = insertObject(containerName, blobName, objMeta); + } catch (AzuException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + azuParameter.getResponse().setStatus(HttpServletResponse.SC_CREATED); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/api/azure/SharedKeyAuth.java b/core/src/com/pspace/ifs/ksan/gw/api/azure/SharedKeyAuth.java new file mode 100644 index 00000000..29c19cd0 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/api/azure/SharedKeyAuth.java @@ -0,0 +1,86 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.api.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.S3UserManager; +import com.pspace.ifs.ksan.libs.PrintStack; + +import com.google.common.base.Strings; + +import java.io.Writer; +import java.text.DateFormat; +import java.text.SimpleDateFormat; +import java.util.Date; +import java.util.TimeZone; + +import javax.xml.stream.XMLOutputFactory; +import javax.xml.stream.XMLStreamWriter; + +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletResponse; + +public class SharedKeyAuth extends AzuRequest { + + public SharedKeyAuth(AzuParameter parameter) { + super(parameter); + logger = LoggerFactory.getLogger(ListContainers.class); + } + + @Override + public void process() throws AzuException { + logger.info(AzuConstants.LOG_SHARED_KEY_AUTH_START); + if (Strings.isNullOrEmpty(S3UserManager.getInstance().getUserByName(azuParameter.getUserName()).getAzureKey())) { + azuParameter.getResponse().setCharacterEncoding(AzuConstants.CHARSET_UTF_8); + try { + XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); + try (Writer writer = azuParameter.getResponse().getWriter()) { + azuParameter.getResponse().setContentType(AzuConstants.CONTENT_TYPE_XML); + XMLStreamWriter xmlStreamWriter = xmlOutputFactory.createXMLStreamWriter(writer); + xmlStreamWriter.writeStartDocument(); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ERROR); + + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CODE); + xmlStreamWriter.writeCharacters(AzuConstants.INVALID_OPERATION); + xmlStreamWriter.writeEndElement(); // End Code + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_MESSAGE); + String message = AzuConstants.INVALID_STORAGE_ACCOUNT + AzuConstants.NEWLINE + AzuConstants.REQUEST_ID + AzuConstants.COLON + + azuParameter.getRequest().getHeader(AzuConstants.X_MS_CLIENT_REQUEST_ID) + AzuConstants.NEWLINE + AzuConstants.TIME + AzuConstants.COLON; + TimeZone tz = TimeZone.getTimeZone(GWConstants.UTC); + DateFormat df = new SimpleDateFormat(AzuConstants.TIME_FORMAT); + df.setTimeZone(tz); + String nowTime = df.format(new Date()); + message += nowTime; + xmlStreamWriter.writeCharacters(message); + xmlStreamWriter.writeEndElement(); // End Message + xmlStreamWriter.writeEndElement(); // End Error + xmlStreamWriter.flush(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } catch (Exception e) { + PrintStack.logging(logger, e); + } + throw new AzuException(AzuErrorCode.INVALID_ACCOUNT, azuParameter); + } else { + azuParameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/condition/DateEquals.java b/core/src/com/pspace/ifs/ksan/gw/condition/DateEquals.java new file mode 100644 index 00000000..bf5293a9 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/condition/DateEquals.java @@ -0,0 +1,128 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.condition; + +import java.util.ArrayList; +import java.util.Iterator; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.ArrayNode; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.format.Tagging; +import com.pspace.ifs.ksan.gw.format.Tagging.TagSet.Tag; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.libs.PrintStack; + +import org.slf4j.LoggerFactory; + +public class DateEquals extends PolicyCondition { + public DateEquals(JsonNode jsonNode) { + super(jsonNode); + logger = LoggerFactory.getLogger(DateEquals.class); + } + + @Override + public void process() throws GWException { + this.value = new ArrayList(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for(int i=0; i compdate) { + ret = true; + break; + } + } + + return ret; + } + + @Override + public boolean compareTagging(S3Parameter s3Parameter) throws GWException { + boolean conditioncheck = false; + String tagkey = key.replace(GWConstants.KEY_EXISTING_OBJECT_TAG + GWConstants.SLASH, GWConstants.EMPTY_STRING); + + String tagvalue = null; + for (String v : value) { + logger.info(v); + tagvalue = v; + break; + } + + if (!Strings.isNullOrEmpty(s3Parameter.getTaggingInfo()) ) { + try { + Tagging tagging = new XmlMapper().readValue(s3Parameter.getTaggingInfo(), Tagging.class); + + if (tagging != null) { + if (tagging.tagset != null && tagging.tagset.tags != null) { + for (Tag t : tagging.tagset.tags) { + logger.info(GWConstants.LOG_T_KEY, tagkey); + logger.info(GWConstants.LOG_T_VALUE, tagvalue); + if (t.key.equals(tagkey) && t.value.equals(tagvalue)) { + conditioncheck = true; + break; + } + } + } + } + } catch (JsonMappingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } else { + conditioncheck = true; + } + + return conditioncheck; + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/condition/DateLessThanEquals.java b/core/src/com/pspace/ifs/ksan/gw/condition/DateLessThanEquals.java new file mode 100644 index 00000000..65c37faa --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/condition/DateLessThanEquals.java @@ -0,0 +1,128 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.condition; + +import java.util.ArrayList; +import java.util.Iterator; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.ArrayNode; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.format.Tagging; +import com.pspace.ifs.ksan.gw.format.Tagging.TagSet.Tag; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.libs.PrintStack; + +import org.slf4j.LoggerFactory; + +public class DateLessThanEquals extends PolicyCondition { + public DateLessThanEquals(JsonNode jsonNode) { + super(jsonNode); + logger = LoggerFactory.getLogger(DateLessThanEquals.class); + } + + @Override + public void process() throws GWException { + this.value = new ArrayList(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i= compdate) { + ret = true; + break; + } + } + + return ret; + } + + @Override + public boolean compareTagging(S3Parameter s3Parameter) throws GWException { + boolean conditioncheck = false; + String tagkey = key.replace(GWConstants.KEY_EXISTING_OBJECT_TAG + GWConstants.SLASH, GWConstants.EMPTY_STRING); + + String tagvalue = null; + for (String v : value) { + logger.info(v); + tagvalue = v; + break; + } + + if (!Strings.isNullOrEmpty(s3Parameter.getTaggingInfo()) ) { + try { + Tagging tagging = new XmlMapper().readValue(s3Parameter.getTaggingInfo(), Tagging.class); + + if (tagging != null) { + if (tagging.tagset != null && tagging.tagset.tags != null) { + for (Tag t : tagging.tagset.tags) { + logger.info(GWConstants.LOG_T_KEY, tagkey); + logger.info(GWConstants.LOG_T_VALUE, tagvalue); + if (t.key.equals(tagkey) && t.value.equals(tagvalue)) { + conditioncheck = true; + break; + } + } + } + } + } catch (JsonMappingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } else { + conditioncheck = true; + } + + return conditioncheck; + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/condition/DateNotEquals.java b/core/src/com/pspace/ifs/ksan/gw/condition/DateNotEquals.java new file mode 100644 index 00000000..d4e74b11 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/condition/DateNotEquals.java @@ -0,0 +1,128 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.condition; + +import java.util.ArrayList; +import java.util.Iterator; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.ArrayNode; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.format.Tagging; +import com.pspace.ifs.ksan.gw.format.Tagging.TagSet.Tag; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.libs.PrintStack; + +import org.slf4j.LoggerFactory; + +public class DateNotEquals extends PolicyCondition { + public DateNotEquals(JsonNode jsonNode) { + super(jsonNode); + logger = LoggerFactory.getLogger(DateNotEquals.class); + } + + @Override + public void process() throws GWException { + this.value = new ArrayList(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info("key : " + this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for(int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for(int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i Double.parseDouble(comp)) { + ret = true; + break; + } + } + + return ret; + } + + @Override + public boolean compareTagging(S3Parameter s3Parameter) throws GWException { + boolean conditioncheck = false; + String tagkey = key.replace(GWConstants.KEY_EXISTING_OBJECT_TAG + GWConstants.SLASH, GWConstants.EMPTY_STRING); + + String tagvalue = null; + for (String v : value) { + logger.info(v); + tagvalue = v; + break; + } + + if (!Strings.isNullOrEmpty(s3Parameter.getTaggingInfo()) ) { + try { + Tagging tagging = new XmlMapper().readValue(s3Parameter.getTaggingInfo(), Tagging.class); + + if (tagging != null) { + if (tagging.tagset != null && tagging.tagset.tags != null) { + for (Tag t : tagging.tagset.tags) { + logger.info(GWConstants.LOG_T_KEY, tagkey); + logger.info(GWConstants.LOG_T_VALUE, tagvalue); + if (t.key.equals(tagkey) && t.value.equals(tagvalue)) { + conditioncheck = true; + break; + } + } + } + } + } catch (JsonMappingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } else { + conditioncheck = true; + } + + return conditioncheck; + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/condition/NumericLessThanEquals.java b/core/src/com/pspace/ifs/ksan/gw/condition/NumericLessThanEquals.java new file mode 100644 index 00000000..67a125af --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/condition/NumericLessThanEquals.java @@ -0,0 +1,126 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.condition; + +import java.util.ArrayList; +import java.util.Iterator; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.ArrayNode; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.format.Tagging; +import com.pspace.ifs.ksan.gw.format.Tagging.TagSet.Tag; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.libs.PrintStack; + +import org.slf4j.LoggerFactory; + +public class NumericLessThanEquals extends PolicyCondition { + public NumericLessThanEquals(JsonNode jsonNode) { + super(jsonNode); + logger = LoggerFactory.getLogger(NumericLessThanEquals.class); + } + + @Override + public void process() throws GWException { + this.value = new ArrayList(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i= Double.parseDouble(comp)) { + ret = true; + break; + } + } + + return ret; + } + + @Override + public boolean compareTagging(S3Parameter s3Parameter) throws GWException { + boolean conditioncheck = false; + String tagkey = key.replace(GWConstants.KEY_EXISTING_OBJECT_TAG + GWConstants.SLASH, GWConstants.EMPTY_STRING); + + String tagvalue = null; + for (String v : value) { + logger.info(v); + tagvalue = v; + break; + } + + if (!Strings.isNullOrEmpty(s3Parameter.getTaggingInfo()) ) { + try { + Tagging tagging = new XmlMapper().readValue(s3Parameter.getTaggingInfo(), Tagging.class); + + if (tagging != null) { + if (tagging.tagset != null && tagging.tagset.tags != null) { + for (Tag t : tagging.tagset.tags) { + logger.info(GWConstants.LOG_T_KEY, tagkey); + logger.info(GWConstants.LOG_T_VALUE, tagvalue); + if (t.key.equals(tagkey) && t.value.equals(tagvalue)) { + conditioncheck = true; + break; + } + } + } + } + } catch (JsonMappingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } else { + conditioncheck = true; + } + + return conditioncheck; + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/condition/NumericNotEquals.java b/core/src/com/pspace/ifs/ksan/gw/condition/NumericNotEquals.java new file mode 100644 index 00000000..097890ab --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/condition/NumericNotEquals.java @@ -0,0 +1,126 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.condition; + +import java.util.ArrayList; +import java.util.Iterator; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.ArrayNode; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.format.Tagging; +import com.pspace.ifs.ksan.gw.format.Tagging.TagSet.Tag; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.libs.PrintStack; + +import org.slf4j.LoggerFactory; + +public class NumericNotEquals extends PolicyCondition { + public NumericNotEquals(JsonNode jsonNode) { + super(jsonNode); + logger = LoggerFactory.getLogger(NumericNotEquals.class); + } + + @Override + public void process() throws GWException { + this.value = new ArrayList(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i value; + + public String getKey() { + return key; + } + + public List getValue() { + return this.value; + } + + public abstract void process() throws GWException; + + public abstract boolean compare(String comp) throws GWException; + + public abstract boolean compareTagging(S3Parameter s3Parameter) throws GWException; + + public static final String STRING_EQUALS = "StringEquals"; + public static final String STRING_LIKE = "StringLike"; + public static final String STRING_NOT_LIKE = "StringNotLike"; + public static final String STRING_EQUALS_IGNORE_CASE = "StringEqualsIgnoreCase"; + public static final String STRING_NOT_EQUALS = "StringNotEquals"; + public static final String STRING_NOT_EQUALS_IGNORE_CASE = "StringNotEqualsIgnoreCase"; + public static final String NUMERIC_EQUALS = "NumericEquals"; + public static final String NUMERIC_NOT_EQUALS = "NumericNotEquals"; + public static final String NUMERIC_LESS_THAN = "NumericLessThan"; + public static final String NUMERIC_LESS_THAN_EQUALS = "NumericLessThanEquals"; + public static final String NUMERIC_GREATER_THAN = "NumericGreaterThan"; + public static final String NUMERIC_GREATER_THAN_EQUALS = "NumericGreaterThanEquals"; + public static final String DATE_EQUALS = "DateEquals"; + public static final String DATE_NOT_EQUALS = "DateNotEquals"; + public static final String DATE_LESS_THAN = "DateLessThan"; + public static final String DATE_LESS_THAN_EQUALS = "DateLessThanEquals"; + public static final String DATE_GREATER_THAN = "DateGreaterThan"; + public static final String DATE_GREATER_THAN_EQUALS = "DateGreaterThanEquals"; + public static final String IP_ADDRESS = "IpAddress"; + public static final String NOT_IP_ADDRESS = "NotIpAddress"; + public static final String LOG_UNKNOWN_CONDITION_TYPE = "Unknown Condition Type"; +} diff --git a/core/src/com/pspace/ifs/ksan/gw/condition/PolicyConditionFactory.java b/core/src/com/pspace/ifs/ksan/gw/condition/PolicyConditionFactory.java new file mode 100644 index 00000000..3049b263 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/condition/PolicyConditionFactory.java @@ -0,0 +1,75 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.condition; + +import com.fasterxml.jackson.databind.JsonNode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class PolicyConditionFactory { + private final static Logger logger = LoggerFactory.getLogger(PolicyConditionFactory.class); + + public PolicyConditionFactory() {} + + public static PolicyCondition createPolicyCondition(String category, JsonNode jsonNode) throws GWException { + switch (category) { + case PolicyCondition.STRING_EQUALS: + return new StringEqual(jsonNode); + case PolicyCondition.STRING_LIKE: + return new StringLike(jsonNode); + case PolicyCondition.STRING_NOT_LIKE: + return new StringNotLike(jsonNode); + case PolicyCondition.STRING_EQUALS_IGNORE_CASE: + return new StringEqualsIgnoreCase(jsonNode); + case PolicyCondition.STRING_NOT_EQUALS: + return new StringNotEquals(jsonNode); + case PolicyCondition.STRING_NOT_EQUALS_IGNORE_CASE: + return new StringNotEqualsIgnoreCase(jsonNode); + + case PolicyCondition.NUMERIC_EQUALS: + return new NumericEquals(jsonNode); + case PolicyCondition.NUMERIC_NOT_EQUALS: + return new NumericNotEquals(jsonNode); + case PolicyCondition.NUMERIC_LESS_THAN: + return new NumericLessThan(jsonNode); + case PolicyCondition.NUMERIC_LESS_THAN_EQUALS: + return new NumericLessThanEquals(jsonNode); + case PolicyCondition.NUMERIC_GREATER_THAN: + return new NumericGreaterThan(jsonNode); + case PolicyCondition.NUMERIC_GREATER_THAN_EQUALS: + return new NumericGreaterThanEquals(jsonNode); + + case PolicyCondition.DATE_EQUALS: + return new DateEquals(jsonNode); + case PolicyCondition.DATE_NOT_EQUALS: + return new DateNotEquals(jsonNode); + case PolicyCondition.DATE_LESS_THAN: + return new DateLessThan(jsonNode); + case PolicyCondition.DATE_LESS_THAN_EQUALS: + return new DateLessThanEquals(jsonNode); + case PolicyCondition.DATE_GREATER_THAN: + return new DateGreaterThan(jsonNode); + case PolicyCondition.DATE_GREATER_THAN_EQUALS: + return new DateGreaterThanEquals(jsonNode); + + case PolicyCondition.IP_ADDRESS: + return new IpAddress(jsonNode); + case PolicyCondition.NOT_IP_ADDRESS: + return new IpAddress(jsonNode); + default: + logger.error(PolicyCondition.LOG_UNKNOWN_CONDITION_TYPE); + break; + } + + return null; + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/condition/StringEqual.java b/core/src/com/pspace/ifs/ksan/gw/condition/StringEqual.java new file mode 100644 index 00000000..2271f5f3 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/condition/StringEqual.java @@ -0,0 +1,126 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.condition; + +import java.util.ArrayList; +import java.util.Iterator; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.ArrayNode; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.format.Tagging; +import com.pspace.ifs.ksan.gw.format.Tagging.TagSet.Tag; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.libs.PrintStack; + +import org.slf4j.LoggerFactory; + +public class StringEqual extends PolicyCondition { + public StringEqual(JsonNode jsonNode) { + super(jsonNode); + logger = LoggerFactory.getLogger(StringEqual.class); + } + + @Override + public void process() throws GWException { + this.value = new ArrayList(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if( fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); + if (jsonNode.isObject()) { + Iterator fieldNames = jsonNode.fieldNames(); + if (fieldNames.hasNext()) { + // read key + String fieldName = fieldNames.next(); + this.key = fieldName; + logger.info(GWConstants.LOG_KEY, this.key); + + // read value + JsonNode fieldValue = jsonNode.get(fieldName); + if (fieldValue != null && fieldValue.isArray() ) { + ArrayNode arrayNode = (ArrayNode) fieldValue; + for (int i=0; i(); logger = LoggerFactory.getLogger(DataPutObject.class); hasAclKeyword = false; + // cacheControl = ""; + // contentDisposition = ""; + // contentEncoding = ""; + // contentLanguage = ""; + // contentMD5 = ""; + // contentType = ""; + // expires = ""; + // acl = ""; + // grantFullControl = ""; + // grantRead = ""; + // grantWrite = ""; + // grantReadAcp = ""; + // grantWriteAcp = ""; + // decodedContentLength = ""; + // serverSideEncryption = ""; + // storageClass = ""; + // websiteRedirectLocation = ""; + // serverSideEncryptionCustomerAlgorithm = ""; + // serverSideEncryptionCustomerKey = ""; + // serverSideEncryptionCustomerKeyMD5 = ""; + // serverSideEncryptionAwsKmsKeyId = ""; + // serverSideEncryptionContext = ""; + // serverSideEncryptionBucketKeyEnabled = ""; + // requestPayer = ""; + // tagging = ""; + // objectLockMode = ""; + // objectLockRetainUntilDate = ""; + // objectLockLegalHold = ""; + // expectedBucketOwner = ""; + // versionId = ""; + // replication = ""; + // dr = ""; + // logging = ""; } @Override - public void extract() throws GWException { + public void extract() throws GWException { + cacheControl = s3Parameter.getRequest().getHeader(GWConstants.CACHE_CONTROL); + contentDisposition = s3Parameter.getRequest().getHeader(GWConstants.CONTENT_DISPOSITION); + contentEncoding = s3Parameter.getRequest().getHeader(GWConstants.CONTENT_ENCODING); + contentLanguage = s3Parameter.getRequest().getHeader(GWConstants.CONTENT_LANGUAGE); + contentMD5 = s3Parameter.getRequest().getHeader(GWConstants.CONTENT_MD5); + contentType = s3Parameter.getRequest().getHeader(GWConstants.CONTENT_TYPE); + expires = s3Parameter.getRequest().getHeader(GWConstants.EXPIRES); + acl = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_ACL); + if (!Strings.isNullOrEmpty(acl)) { + hasAclKeyword = true; + } + grantFullControl = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_GRANT_FULL_CONTROL); + if (!Strings.isNullOrEmpty(grantFullControl)) { + hasAclKeyword = true; + } + grantRead = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_GRANT_READ); + if (!Strings.isNullOrEmpty(grantRead)) { + hasAclKeyword = true; + } + grantReadAcp = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_GRANT_READ_ACP); + if (!Strings.isNullOrEmpty(grantReadAcp)) { + hasAclKeyword = true; + } + grantWrite = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_GRANT_WRITE); + if (!Strings.isNullOrEmpty(grantWrite)) { + hasAclKeyword = true; + } + grantWriteAcp = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_GRANT_WRITE_ACP); + if (!Strings.isNullOrEmpty(grantWriteAcp)) { + hasAclKeyword = true; + } + decodedContentLength = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_DECODED_CONTENT_LENGTH); + serverSideEncryption = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION); + storageClass = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_STORAGE_CLASS); + websiteRedirectLocation = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_WEBSITE_REDIRECT_LOCATION); + serverSideEncryptionCustomerAlgorithm = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM); + + serverSideEncryptionCustomerKey = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY); + serverSideEncryptionCustomerKeyMD5 = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5); + serverSideEncryptionAwsKmsKeyId = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID); + serverSideEncryptionContext = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CONTEXT); + serverSideEncryptionBucketKeyEnabled = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_BUCKET_KEY_ENABLED); + requestPayer = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_REQUEST_PAYER); + tagging = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_TAGGING); + objectLockMode = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_OBJECT_LOCK_MODE); + objectLockRetainUntilDate = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE); + objectLockLegalHold = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_OBJECT_LOCK_LEGAL_HOLD); + expectedBucketOwner = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_EXPECTED_BUCKET_OWNER); + versionId = s3Parameter.getRequest().getHeader(GWConstants.X_IFS_VERSION_ID); + for (String headerName : Collections.list(s3Parameter.getRequest().getHeaderNames())) { - if (headerName.equalsIgnoreCase(GWConstants.CACHE_CONTROL)) { - cacheControl = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_DISPOSITION)) { - contentDisposition = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_ENCODING)) { - contentEncoding = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_LANGUAGE)) { - contentLanguage = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_MD5)) { - contentMD5 = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_TYPE)) { - contentType = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.EXPIRES)) { - expires = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_ACL)) { - acl = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_FULL_CONTROL)) { - grantFullControl = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_READ)) { - grantRead = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_READ_ACP)) { - grantReadAcp = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_WRITE)) { - grantWrite = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_WRITE_ACP)) { - grantWriteAcp = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_DECODED_CONTENT_LENGTH)) { - decodedContentLength = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION)) { - serverSideEncryption = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_STORAGE_CLASS)) { - storageClass = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_WEBSITE_REDIRECT_LOCATION)) { - websiteRedirectLocation = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM)) { - serverSideEncryptionCustomerAlgorithm = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY)) { - serverSideEncryptionCustomerKey = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5)) { - serverSideEncryptionCustomerKeyMD5 = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID)) { - serverSideEncryptionAwsKmsKeyId = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CONTEXT)) { - serverSideEncryptionContext = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_BUCKET_KEY_ENABLED)) { - serverSideEncryptionBucketKeyEnabled = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_REQUEST_PAYER)) { - requestPayer = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_TAGGING)) { - tagging = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_OBJECT_LOCK_MODE)) { - objectLockMode = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE)) { - objectLockRetainUntilDate = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_OBJECT_LOCK_LEGAL_HOLD)) { - objectLockLegalHold = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_EXPECTED_BUCKET_OWNER)) { - expectedBucketOwner = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - } else if (headerName.startsWith(GWConstants.USER_METADATA_PREFIX) ) { + if (headerName.startsWith(GWConstants.USER_METADATA_PREFIX) ) { userMetadata.put(headerName, s3Parameter.getRequest().getHeader(headerName)); } else if (headerName.equalsIgnoreCase(GWConstants.X_IFS_VERSION_ID)) { - versionId = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); + versionId = s3Parameter.getRequest().getHeader(headerName); } else if (headerName.equalsIgnoreCase(GWConstants.X_IFS_REPLICATION)) { replication = GWConstants.IFS_HEADER_REPLICATION; } else if (headerName.equalsIgnoreCase(GWConstants.X_IFS_DR)) { @@ -144,6 +163,110 @@ public void extract() throws GWException { logging = GWConstants.IFS_HEADER_LOGGING; } } + + // for (String headerName : Collections.list(s3Parameter.getRequest().getHeaderNames())) { + // if (headerName.equalsIgnoreCase(GWConstants.CACHE_CONTROL)) { + // cacheControl = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_DISPOSITION)) { + // contentDisposition = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_ENCODING)) { + // contentEncoding = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_LANGUAGE)) { + // contentLanguage = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_MD5)) { + // contentMD5 = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.CONTENT_TYPE)) { + // contentType = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.EXPIRES)) { + // expires = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_ACL)) { + // acl = s3Parameter.getRequest().getHeader(headerName); + // if (!Strings.isNullOrEmpty(acl)) { + // hasAclKeyword = true; + // } + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_FULL_CONTROL)) { + // grantFullControl = s3Parameter.getRequest().getHeader(headerName); + // if (!Strings.isNullOrEmpty(grantFullControl)) { + // hasAclKeyword = true; + // } + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_READ)) { + // grantRead = s3Parameter.getRequest().getHeader(headerName); + // if (!Strings.isNullOrEmpty(grantRead)) { + // hasAclKeyword = true; + // } + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_READ_ACP)) { + // grantReadAcp = s3Parameter.getRequest().getHeader(headerName); + // if (!Strings.isNullOrEmpty(grantReadAcp)) { + // hasAclKeyword = true; + // } + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_WRITE)) { + // grantWrite = s3Parameter.getRequest().getHeader(headerName); + // if (!Strings.isNullOrEmpty(grantWrite)) { + // hasAclKeyword = true; + // } + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_WRITE_ACP)) { + // grantWriteAcp = s3Parameter.getRequest().getHeader(headerName); + // if (!Strings.isNullOrEmpty(grantWriteAcp)) { + // hasAclKeyword = true; + // } + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_DECODED_CONTENT_LENGTH)) { + // decodedContentLength = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION)) { + // serverSideEncryption = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_STORAGE_CLASS)) { + // storageClass = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_WEBSITE_REDIRECT_LOCATION)) { + // websiteRedirectLocation = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM)) { + // serverSideEncryptionCustomerAlgorithm = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY)) { + // serverSideEncryptionCustomerKey = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5)) { + // serverSideEncryptionCustomerKeyMD5 = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID)) { + // serverSideEncryptionAwsKmsKeyId = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CONTEXT)) { + // serverSideEncryptionContext = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_BUCKET_KEY_ENABLED)) { + // serverSideEncryptionBucketKeyEnabled = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_REQUEST_PAYER)) { + // requestPayer = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_TAGGING)) { + // tagging = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_OBJECT_LOCK_MODE)) { + // objectLockMode = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE)) { + // objectLockRetainUntilDate = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_OBJECT_LOCK_LEGAL_HOLD)) { + // objectLockLegalHold = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_EXPECTED_BUCKET_OWNER)) { + // expectedBucketOwner = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.startsWith(GWConstants.USER_METADATA_PREFIX) ) { + // userMetadata.put(headerName, s3Parameter.getRequest().getHeader(headerName)); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_IFS_VERSION_ID)) { + // versionId = s3Parameter.getRequest().getHeader(headerName); + // } else if (headerName.equalsIgnoreCase(GWConstants.X_IFS_REPLICATION)) { + // replication = GWConstants.IFS_HEADER_REPLICATION; + // } else if (headerName.equalsIgnoreCase(GWConstants.X_IFS_DR)) { + // dr = GWConstants.IFS_HEADER_DR; + // } else if (headerName.equalsIgnoreCase(GWConstants.X_IFS_LOGGING)) { + // logging = GWConstants.IFS_HEADER_LOGGING; + // } + // } + + keyXAmzAcl = getAcl(); + keyXAmzGrantFullControl = getGrantFullControl(); + keyXAmzGrantRead = getGrantRead(); + keyXAmzGrantReadAcp = getGrantReadAcp(); + keyXAmzGrantWrite = getGrantWrite(); + keyXAmzGrantWriteAcp = getGrantWriteAcp(); + keyXAmzServerSideEncryption = getServerSideEncryption(); + keyXAmzServerSideEncryptionAwsKmsKeyId = getServerSideEncryptionAwsKmsKeyId(); + keyXAmzStorageClass = getStorageClass(); + keyXAmzWebsiteRedirectLocation = getWebsiteRedirectLocation(); + keyXAmzObjectLockMode = getObjectLockMode(); + keyXAmzObjectLockRetainUntilDate = getObjectLockRetainUntilDate(); + keyXAmzObjectLockLegalHold = getObjectLockLegalHold(); } public boolean hasAclKeyword() { diff --git a/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectAcl.java b/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectAcl.java index 471cf287..67f34299 100644 --- a/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectAcl.java +++ b/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectAcl.java @@ -50,29 +50,49 @@ public void extract() throws GWException { if (headerName.equalsIgnoreCase(GWConstants.CONTENT_MD5)) { contentMd5 = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_ACL)) { - hasAclKeyword = true; acl = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); + if (!Strings.isNullOrEmpty(acl)) { + hasAclKeyword = true; + } } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_FULL_CONTROL)) { grantFullControl = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; + if (!Strings.isNullOrEmpty(grantFullControl)) { + hasAclKeyword = true; + } } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_READ)) { grantRead = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; + if (!Strings.isNullOrEmpty(grantRead)) { + hasAclKeyword = true; + } } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_READ_ACP)) { grantReadAcp = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; + if (!Strings.isNullOrEmpty(grantReadAcp)) { + hasAclKeyword = true; + } } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_WRITE)) { grantWrite = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; + if (!Strings.isNullOrEmpty(grantWrite)) { + hasAclKeyword = true; + } } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_GRANT_WRITE_ACP)) { grantWriteAcp = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); - hasAclKeyword = true; + if (!Strings.isNullOrEmpty(grantWriteAcp)) { + hasAclKeyword = true; + } } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_REQUEST_PAYER)) { requestPayer = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_EXPECTED_BUCKET_OWNER)) { expectedBucketOwner = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); } } + + keyXAmzAcl = getAcl(); + keyXAmzGrantFullControl = getGrantFullControl(); + keyXAmzGrantRead = getGrantRead(); + keyXAmzGrantReadAcp = getGrantReadAcp(); + keyXAmzGrantWrite = getGrantWrite(); + keyXAmzGrantWriteAcp = getGrantWriteAcp(); + keyVersionId = getVersionId(); } public boolean hasAclKeyword() { diff --git a/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectLegalHold.java b/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectLegalHold.java new file mode 100644 index 00000000..0051bed7 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectLegalHold.java @@ -0,0 +1,73 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.data; + +import java.util.Collections; + +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.utils.GWConstants; + +import org.slf4j.LoggerFactory; + +public class DataPutObjectLegalHold extends S3DataRequest { + private String versionId; + + private String contentMd5; + private String requestPayer; + private String expectedBucketOwner; + private String legalHoldXml; + + public DataPutObjectLegalHold(S3Parameter s3Parameter) throws GWException { + super(s3Parameter); + logger = LoggerFactory.getLogger(DataPutObjectLegalHold.class); + } + + @Override + public void extract() throws GWException { + versionId = s3Parameter.getRequest().getParameter(GWConstants.VERSION_ID); + if (Strings.isNullOrEmpty(versionId)) { + logger.debug(GWConstants.LOG_DATA_VERSION_ID_NULL); + } + + for (String headerName : Collections.list(s3Parameter.getRequest().getHeaderNames())) { + if (headerName.equalsIgnoreCase(GWConstants.CONTENT_MD5)) { + contentMd5 = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); + } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_REQUEST_PAYER)) { + requestPayer = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); + } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_EXPECTED_BUCKET_OWNER)) { + expectedBucketOwner = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); + } + } + } + + public String getVersionId() { + return versionId; + } + + public String getContentMd5() { + return contentMd5; + } + + public String getRequestPayer() { + return requestPayer; + } + + public String getExpectedBucketOwner() { + return expectedBucketOwner; + } + + public String getLegalHoldXml() throws GWException { + legalHoldXml = readXml(); + return legalHoldXml; + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectRetention.java b/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectRetention.java index 7ec6f02d..d8658d40 100644 --- a/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectRetention.java +++ b/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectRetention.java @@ -1,3 +1,13 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ package com.pspace.ifs.ksan.gw.data; import java.util.Collections; diff --git a/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectTagging.java b/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectTagging.java index 606420c5..975a3498 100644 --- a/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/data/DataPutObjectTagging.java @@ -48,6 +48,8 @@ public void extract() throws GWException { expectedBucketOwner = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); } } + + keyVersionId = getVersionId(); } public String getVersionId() { diff --git a/core/src/com/pspace/ifs/ksan/gw/data/DataRestoreObject.java b/core/src/com/pspace/ifs/ksan/gw/data/DataRestoreObject.java new file mode 100644 index 00000000..2c400455 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/DataRestoreObject.java @@ -0,0 +1,72 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.data; + +import java.util.Collections; + +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.gw.utils.GWConstants; + +import org.slf4j.LoggerFactory; + +public class DataRestoreObject extends S3DataRequest { + private String versionId; + private String requestPayer; + private String expectedBucketOwner; + private String sdkChecksumAlgorithm; + private String restoreXml; + + public DataRestoreObject(S3Parameter s3Parameter) throws GWException { + super(s3Parameter); + logger = LoggerFactory.getLogger(DataRestoreObject.class); + } + + @Override + public void extract() throws GWException { + versionId = s3Parameter.getRequest().getParameter(GWConstants.VERSION_ID); + if (Strings.isNullOrEmpty(versionId)) { + logger.debug(GWConstants.LOG_DATA_VERSION_ID_NULL); + } + + for (String headerName : Collections.list(s3Parameter.getRequest().getHeaderNames())) { + if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_REQUEST_PAYER)) { + requestPayer = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); + } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_EXPECTED_BUCKET_OWNER)) { + expectedBucketOwner = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); + } else if (headerName.equalsIgnoreCase(GWConstants.X_AMZ_SDK_CHECKSUM_ALGORITHM)) { + sdkChecksumAlgorithm = Strings.nullToEmpty(s3Parameter.getRequest().getHeader(headerName)); + } + } + } + + public String getVersionId() { + return versionId; + } + + public String getSdkChecksumAlgorithm() { + return sdkChecksumAlgorithm; + } + + public String getRequestPayer() { + return requestPayer; + } + + public String getExpectedBucketOwner() { + return expectedBucketOwner; + } + + public String getRetoreXml() throws GWException { + restoreXml = readXml(); + return restoreXml; + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/data/S3DataRequest.java b/core/src/com/pspace/ifs/ksan/gw/data/S3DataRequest.java index 21611eb2..bdb4e2cf 100644 --- a/core/src/com/pspace/ifs/ksan/gw/data/S3DataRequest.java +++ b/core/src/com/pspace/ifs/ksan/gw/data/S3DataRequest.java @@ -24,10 +24,33 @@ public abstract class S3DataRequest { - String contentLength; + protected String contentLength; + protected Logger logger; protected S3Parameter s3Parameter; + protected String keyDelimiter; + protected String keyMaxKeys; + protected String keyPrefix; + + // policy key putobject + protected String keyXAmzAcl; + protected String keyXAmzCopySource; + protected String keyXAmzGrantFullControl; + protected String keyXAmzGrantRead; + protected String keyXAmzGrantReadAcp; + protected String keyXAmzGrantWrite; + protected String keyXAmzGrantWriteAcp; + protected String keyXAmzMetadataDirective; + protected String keyXAmzServerSideEncryption; + protected String keyXAmzServerSideEncryptionAwsKmsKeyId; + protected String keyXAmzStorageClass; + protected String keyXAmzWebsiteRedirectLocation; + protected String keyXAmzObjectLockMode; + protected String keyXAmzObjectLockRetainUntilDate; + protected String keyXAmzObjectLockRemainingRetentionDays; + protected String keyXAmzObjectLockLegalHold; + public S3DataRequest(S3Parameter s3Parameter) throws GWException { this.s3Parameter = s3Parameter; checkContentLength(); @@ -105,4 +128,87 @@ protected String readJson() throws GWException { } public abstract void extract() throws GWException; + + public String getPolicyDelimter() { + return keyDelimiter; + } + + public String getPolicyMaxkeys() { + return keyMaxKeys; + } + + public String getPolicyPrefix() { + return keyPrefix; + } + + public String getXAmzAcl() { + return keyXAmzAcl; + } + + public String getXAmzCopySource() { + return keyXAmzCopySource; + } + + public String getXAmzGrantFullControl() { + return keyXAmzGrantFullControl; + } + + public String getXAmzGrantRead() { + return keyXAmzGrantRead; + } + + public String getXAmzGrantReadAcp() { + return keyXAmzGrantReadAcp; + } + + public String getXAmzGrantWrite() { + return keyXAmzGrantWrite; + } + + public String getXAmzGrantWriteAcp() { + return keyXAmzGrantWriteAcp; + } + + public String getXAmzMetadataDirective() { + return keyXAmzMetadataDirective; + } + + public String getXAmzServerSideEncryption() { + return keyXAmzServerSideEncryption; + } + + public String getXAmzServerSideEncryptionAwsKmsKeyId() { + return keyXAmzServerSideEncryptionAwsKmsKeyId; + } + + public String getXAmzStorageClass() { + return keyXAmzStorageClass; + } + + public String getXAmzWebsiteRedirectLocation() { + return keyXAmzWebsiteRedirectLocation; + } + + public String getXAmzObjectLockMode() { + return keyXAmzObjectLockMode; + } + + public String getXAmzObjectLockRetainUntilDate() { + return keyXAmzObjectLockRetainUntilDate; + } + + public String getXAmzObjectLockRemainingRetentionDays() { + return keyXAmzObjectLockRemainingRetentionDays; + } + + public String getXAmzObjectLockLegalHold() { + return keyXAmzObjectLockLegalHold; + } + + // policy key delete + protected String keyVersionId; + + public String getVersionId() { + return keyVersionId; + } } diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/AzuDataRequest.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/AzuDataRequest.java new file mode 100644 index 00000000..91076052 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/AzuDataRequest.java @@ -0,0 +1,109 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import java.io.IOException; + +import org.slf4j.Logger; +import com.google.common.base.Strings; +import com.google.common.net.HttpHeaders; +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.libs.PrintStack; + +import com.pspace.ifs.ksan.gw.utils.AzuConstants; + +public abstract class AzuDataRequest { + String contentLength; + protected Logger logger; + protected AzuParameter parameter; + + public AzuDataRequest(AzuParameter parameter) throws AzuException { + this.parameter = parameter; + checkContentLength(); + if (parameter.getResponse() != null) { + for (String header : parameter.getResponse().getHeaderNames()) { + parameter.addResponseSize(header.length()); + String value = parameter.getResponse().getHeader(header); + if (!Strings.isNullOrEmpty(value)) { + parameter.addResponseSize(value.length()); + } + } + } + } + + private void checkContentLength() throws AzuException { + contentLength = parameter.getRequest().getHeader(HttpHeaders.CONTENT_LENGTH); + if (!Strings.isNullOrEmpty(contentLength)) { + long length = Long.parseLong(contentLength); + if (length < 0) { + throw new AzuException(AzuErrorCode.INVALID_ARGUMENT, parameter); + } + } + } + + protected String getContentLength() { + return contentLength; + } + + protected String readXml() throws AzuException { + String ret = null; + + try { + byte[] xml = parameter.getInputStream().readAllBytes(); + parameter.addRequestSize(xml.length); + ret = new String(xml); + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.INTERNAL_SERVER_ERROR, parameter); + } + + logger.info(ret); + + if(Strings.isNullOrEmpty(ret)) { + logger.warn(AzuErrorCode.INVALID_ARGUMENT.getMessage()); + throw new AzuException(AzuErrorCode.INVALID_ARGUMENT, parameter); + } + + if (!ret.contains(AzuConstants.XML_VERSION)) { + ret = AzuConstants.XML_VERSION_FULL + ret; + } + + return ret; + } + + protected String readJson() throws AzuException { + String ret = null; + + try { + byte[] json = parameter.getInputStream().readAllBytes(); + parameter.addRequestSize(json.length); + ret = new String(json); + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.INTERNAL_SERVER_ERROR, parameter); + } + + logger.info(ret); + + if(Strings.isNullOrEmpty(ret)) { + logger.warn(AzuErrorCode.INVALID_ARGUMENT.getMessage()); + throw new AzuException(AzuErrorCode.INVALID_ARGUMENT, parameter); + } + + return ret; + } + + public abstract void extract() throws AzuException; +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/DataCreateContainer.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataCreateContainer.java new file mode 100644 index 00000000..2ecd98a7 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataCreateContainer.java @@ -0,0 +1,34 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; + +public class DataCreateContainer extends AzuDataRequest { + + public DataCreateContainer(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataCreateContainer.class); + } + + @Override + public void extract() throws AzuException { + // TODO Auto-generated method stub + + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/DataDeleteBlob.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataDeleteBlob.java new file mode 100644 index 00000000..93ea2f80 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataDeleteBlob.java @@ -0,0 +1,34 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; + +public class DataDeleteBlob extends AzuDataRequest { + + public DataDeleteBlob(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataCreateContainer.class); + } + + @Override + public void extract() throws AzuException { + // TODO Auto-generated method stub + + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/DataDeleteContainer.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataDeleteContainer.java new file mode 100644 index 00000000..9a7f9b15 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataDeleteContainer.java @@ -0,0 +1,34 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; + +public class DataDeleteContainer extends AzuDataRequest { + + public DataDeleteContainer(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataDeleteContainer.class); + } + + @Override + public void extract() throws AzuException { + // TODO Auto-generated method stub + + } + +} + diff --git a/core/common/common/ResponseData.py b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataGetBlob.java similarity index 56% rename from core/common/common/ResponseData.py rename to core/src/com/pspace/ifs/ksan/gw/data/azure/DataGetBlob.java index ad424978..3f38f683 100644 --- a/core/common/common/ResponseData.py +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataGetBlob.java @@ -1,4 +1,4 @@ -""" +/* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version @@ -7,22 +7,28 @@ * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" +*/ -import json -from const.common import RetKeyResult, RetKeyCode, RetKeyMessage +package com.pspace.ifs.ksan.gw.data.azure; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; -from Enums.EnumResponseResult import EnumResponseResult +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +public class DataGetBlob extends AzuDataRequest { -class ResponseData: - def __init__(self, result: EnumResponseResult = EnumResponseResult.Error, code: str = "", messsage: str = "") -> None: - self.Result = result - self.Code = code - self.Message = messsage + public DataGetBlob(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataGetBlob.class); + } + + @Override + public void extract() throws AzuException { + // TODO Auto-generated method stub + + } + +} - def load(self, data: dict): - self.Result = data[RetKeyResult] - self.Code = data[RetKeyCode] - self.Message = data[RetKeyMessage] diff --git a/core/common/common/ResponseMqData.py b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataHeadBlob.java similarity index 56% rename from core/common/common/ResponseMqData.py rename to core/src/com/pspace/ifs/ksan/gw/data/azure/DataHeadBlob.java index fa7d9345..600c8c6b 100644 --- a/core/common/common/ResponseMqData.py +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataHeadBlob.java @@ -1,4 +1,4 @@ -""" +/* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version @@ -7,22 +7,28 @@ * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" +*/ -import json +package com.pspace.ifs.ksan.gw.data.azure; -from Enums.EnumResponseResult import EnumResponseResult -from const.common import RetKeyResult, RetKeyCode, RetKeyMessage +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; -class ResponseMqData: - def __init__(self, result: EnumResponseResult = EnumResponseResult.Error, code: str = "", messsage: str = "") -> None: - self.Result = result - self.Code = code - self.Message = messsage - self.IsProcessed = False +public class DataHeadBlob extends AzuDataRequest { + + public DataHeadBlob(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataHeadBlob.class); + } + + @Override + public void extract() throws AzuException { + // TODO Auto-generated method stub + + } + +} - def load(self, data: dict): - self.Result = data[RetKeyResult] - self.Code = data[RetKeyCode] - self.Message = data[RetKeyMessage] diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/DataHeadContainer.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataHeadContainer.java new file mode 100644 index 00000000..326e6ff9 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataHeadContainer.java @@ -0,0 +1,34 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; + +public class DataHeadContainer extends AzuDataRequest { + + public DataHeadContainer(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataHeadContainer.class); + } + + @Override + public void extract() throws AzuException { + // TODO Auto-generated method stub + + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/DataListBlobs.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataListBlobs.java new file mode 100644 index 00000000..ef991fdb --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataListBlobs.java @@ -0,0 +1,56 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; + +public class DataListBlobs extends AzuDataRequest { + String prefix; + String delimiter; + String maxResults; + String include; + + public DataListBlobs(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataHeadContainer.class); + } + + @Override + public void extract() throws AzuException { + prefix = parameter.getRequest().getParameter(AzuConstants.PARAMETER_PREFIX); + delimiter = parameter.getRequest().getParameter(AzuConstants.PARAMETER_DELIMITER); + maxResults = parameter.getRequest().getParameter(AzuConstants.PARAMETER_MAX_RESULTS); + include = parameter.getRequest().getParameter(AzuConstants.PARAMETER_INCLUDE); + } + + public String getPrefix() { + return prefix; + } + + public String getDelimiter() { + return delimiter; + } + + public String getMaxResults() { + return maxResults; + } + + public String getInclude() { + return include; + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/DataListContainers.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataListContainers.java new file mode 100644 index 00000000..4d913c05 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataListContainers.java @@ -0,0 +1,34 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; + +public class DataListContainers extends AzuDataRequest { + + public DataListContainers(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataHeadContainer.class); + } + + @Override + public void extract() throws AzuException { + // TODO Auto-generated method stub + + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/DataPutBlob.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataPutBlob.java new file mode 100644 index 00000000..ab285353 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataPutBlob.java @@ -0,0 +1,60 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import java.util.Collections; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.GWConstants; + +public class DataPutBlob extends AzuDataRequest { + private String contentType; + private String contentMD5; + + public DataPutBlob(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataPutBlob.class); + contentType = ""; + contentMD5 = ""; + } + + @Override + public String getContentLength() { + return super.getContentLength(); + } + + @Override + public void extract() throws AzuException { + for (String headerName : Collections.list(parameter.getRequest().getHeaderNames())) { + if (headerName.equalsIgnoreCase(AzuConstants.X_MS_BLOB_CONTENT_TYPE)) { + contentType = Strings.nullToEmpty(parameter.getRequest().getHeader(headerName)); + } else if (headerName.equalsIgnoreCase(AzuConstants.X_MS_BLOB_CONTENT_MD5)) { + contentMD5 = Strings.nullToEmpty(parameter.getRequest().getHeader(headerName)); + } + } + } + + public String getContentType() { + return contentType; + } + + public String getContentMD5() { + return contentMD5; + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/DataPutBlock.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataPutBlock.java new file mode 100644 index 00000000..982e9964 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataPutBlock.java @@ -0,0 +1,46 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import java.util.Collections; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; + +public class DataPutBlock extends AzuDataRequest { + private String blockid; + + public DataPutBlock(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataPutBlock.class); + } + + @Override + public void extract() throws AzuException { + blockid = parameter.getRequest().getParameter(AzuConstants.BLOCKID); + } + + @Override + public String getContentLength() { + return super.getContentLength(); + } + + public String getBlockId() { + return blockid; + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/data/azure/DataPutBlockList.java b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataPutBlockList.java new file mode 100644 index 00000000..78764fa9 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/data/azure/DataPutBlockList.java @@ -0,0 +1,74 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ + +package com.pspace.ifs.ksan.gw.data.azure; + +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; + +import java.util.Collections; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; + +public class DataPutBlockList extends AzuDataRequest { + + private String blockid; + private String blobContentType; + private String blobContentMD5; + + public DataPutBlockList(AzuParameter parameter) throws AzuException { + super(parameter); + logger = LoggerFactory.getLogger(DataPutBlockList.class); + } + + @Override + public void extract() throws AzuException { + // TODO Auto-generated method stub + blockid = parameter.getRequest().getParameter(AzuConstants.BLOCKID); + + for (String headerName : Collections.list(parameter.getRequest().getHeaderNames())) { + if (headerName.equalsIgnoreCase(AzuConstants.X_MS_BLOB_CONTENT_TYPE)) { + blobContentType = Strings.nullToEmpty(parameter.getRequest().getHeader(headerName)); + } else if (headerName.equalsIgnoreCase(AzuConstants.X_MS_BLOB_CONTENT_MD5)) { + blobContentMD5 = Strings.nullToEmpty(parameter.getRequest().getHeader(headerName)); + } + } + } + + @Override + public String getContentLength() { + return super.getContentLength(); + } + + public String getBlockId() { + return blockid; + } + + public String getBlobContentType() { + return blobContentType; + } + + public String getBlobContentMD5() { + return blobContentMD5; + } + + public String getXml() throws AzuException { + return readXml(); + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/exception/AzuErrorCode.java b/core/src/com/pspace/ifs/ksan/gw/exception/AzuErrorCode.java new file mode 100644 index 00000000..b8e08ad0 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/exception/AzuErrorCode.java @@ -0,0 +1,147 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.exception; + +import static java.util.Objects.requireNonNull; + +import com.google.common.base.CaseFormat; + +import jakarta.servlet.http.HttpServletResponse; + +public enum AzuErrorCode { + SERVER_ERROR(HttpServletResponse.SC_INTERNAL_SERVER_ERROR, "Internal Server Error"), + ACCESS_DENIED(HttpServletResponse.SC_FORBIDDEN, "Forbidden"), + BAD_DIGEST(HttpServletResponse.SC_BAD_REQUEST, "Bad Digest"), + BAD_REQUEST(HttpServletResponse.SC_BAD_REQUEST, "Bad Request"), + INVALID_ACCOUNT(HttpServletResponse.SC_BAD_REQUEST, "Invalid storage account."), + CONTAINER_ALREADY_EXISTS(HttpServletResponse.SC_CONFLICT, + "The requested container name is not available." + + " The container namespace is shared by all users of the system." + + " Please select a different name and try again."), + BUCKET_ALREADY_OWNED_BY_YOU(HttpServletResponse.SC_CONFLICT, + "Your previous request to create the named bucket" + + " succeeded and you already own it."), + BUCKET_NOT_EMPTY(HttpServletResponse.SC_CONFLICT, + "The bucket you tried to delete is not empty"), + ENTITY_TOO_SMALL(HttpServletResponse.SC_BAD_REQUEST, + "Your proposed upload is smaller than the minimum allowed object size."), + ENTITY_TOO_LARGE(HttpServletResponse.SC_BAD_REQUEST, + "Your proposed upload exceeds the maximum allowed object size."), + INVALID_STORAGE_ACCOUNT(HttpServletResponse.SC_BAD_REQUEST, "Invalid storage account."), + INVALID_ACCESS_KEY_ID(HttpServletResponse.SC_FORBIDDEN, "Forbidden"), + INVALID_ARGUMENT(HttpServletResponse.SC_BAD_REQUEST, "Bad Request"), + INVALID_BUCKET_NAME(HttpServletResponse.SC_BAD_REQUEST, + "The specified bucket is not valid."), + INVALID_BUCKET_STATE(HttpServletResponse.SC_CONFLICT, + "The specified bucket is not valid."), + INVALID_DIGEST(HttpServletResponse.SC_BAD_REQUEST, "Invalid Digest"), + INVALID_LOCATION_CONSTRAINT(HttpServletResponse.SC_BAD_REQUEST, + "The specified location constraint is not valid. For" + + " more information about Regions, see How to Select" + + " a Region for Your Buckets."), + INVALID_RANGE(HttpServletResponse.SC_REQUESTED_RANGE_NOT_SATISFIABLE, + "The requested range is not satisfiable"), + INVALID_PART(HttpServletResponse.SC_BAD_REQUEST, + "One or more of the specified parts could not be found." + + " The part may not have been uploaded, or the specified entity" + + " tag may not match the part's entity tag."), + INVALID_REQUEST(HttpServletResponse.SC_BAD_REQUEST, "Bad Request"), + INVALID_TAG(HttpServletResponse.SC_BAD_REQUEST, + "The requested tagging key or value exceeds length"), + INVALID_EXCEED_TAG(HttpServletResponse.SC_BAD_REQUEST, + "The requested tagging count has not exceed 10"), + INVALID_CORS_ORIGIN(HttpServletResponse.SC_BAD_REQUEST, + "Insufficient information. Origin request header needed."), + INVALID_CORS_METHOD(HttpServletResponse.SC_BAD_REQUEST, + "The specified Access-Control-Request-Method is not valid."), + MALFORMED_X_M_L(HttpServletResponse.SC_BAD_REQUEST, + "The XML you provided was not well-formed or did not validate" + + " against our published schema."), + MAX_MESSAGE_LENGTH_EXCEEDED(HttpServletResponse.SC_BAD_REQUEST, + "Your request was too big."), + METHOD_NOT_ALLOWED(HttpServletResponse.SC_METHOD_NOT_ALLOWED, + "Method Not Allowed"), + MISSING_CONTENT_LENGTH(HttpServletResponse.SC_LENGTH_REQUIRED, + "Length Required"), + NO_SUCH_CONTAINER(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not exist"), + NO_SUCH_WEBSITE_CONFIGURATION(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not have a website configuration"), + NO_SUCH_POLICY_CONFIGURATION(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not have a policy configuration"), + NO_SUCH_CORS_CONFIGURATION(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not have a CORS configuration"), + NO_SUCH_TAG_SET_ERROR(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not have a Tagging SET"), + NO_SUCH_LIFECYCLE_CONFIGURATION(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not have a LifeCycle configuration"), + NO_SUCH_ENCRYPTION_CONFIGURATION(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not have a Server Side Encryption configuration"), + NO_SUCH_PUBLICACCESSBLOCK_CONFIGURATION(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not have a Public Access Block configuration"), + NO_SUCH_REPLICATION_CONFIGURATION(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not have a Replication configuration"), + OBJECT_LOCK_CONFIGURATION_NOT_FOUND_ERROR(HttpServletResponse.SC_NOT_FOUND, + "The specified bucket does not have a Object Lock configuration"), + ILLEGAL_VERSIONING_CONFIGURATION(HttpServletResponse.SC_BAD_REQUEST, + "Indicates that the versioning configuration specified in the request is invalid."), + INVALID_REPLICATION_REQUEST(HttpServletResponse.SC_BAD_REQUEST, + "Versioning must be Enabled on the bucket to apply a replication configuration"), + NO_SUCH_KEY(HttpServletResponse.SC_NOT_FOUND, + "The specified blob does not exist."), + NO_SUCH_UPLOAD(HttpServletResponse.SC_NOT_FOUND, "Not Found"), + NOT_IMPLEMENTED(HttpServletResponse.SC_NOT_IMPLEMENTED, + "A header you provided implies functionality that is not" + + " implemented."), + INSUFFICIENT_STORAGE(507, "S3 Quota has exceeded"), // 507 INSUFFICIENT_STORAGE + PRECONDITION_FAILED(HttpServletResponse.SC_PRECONDITION_FAILED, + "At least one of the preconditions you specified did not hold."), + REQUEST_TIME_TOO_SKEWED(HttpServletResponse.SC_FORBIDDEN, "Forbidden"), + REQUEST_TIMEOUT(HttpServletResponse.SC_BAD_REQUEST, "Bad Request"), + SIGNATURE_DOES_NOT_MATCH(HttpServletResponse.SC_FORBIDDEN, "Forbidden"), + KEY_DOES_NOT_MATCH(HttpServletResponse.SC_FORBIDDEN, "Forbidden"), + ILLEGAL_RANGE(HttpServletResponse.SC_REQUESTED_RANGE_NOT_SATISFIABLE , "Server cannot serve the requested byte range"), + DOES_NOT_MATCH(HttpServletResponse.SC_NOT_MODIFIED, "not modified"), + X_AMZ_CONTENT_S_H_A_256_MISMATCH(HttpServletResponse.SC_BAD_REQUEST, + "The provided 'x-amz-content-sha256' header does not match what" + + " was computed."), + INTERNAL_SERVER_DB_ERROR(HttpServletResponse.SC_INTERNAL_SERVER_ERROR, "S3 database error has occurred"), + INTERNAL_SERVER_ERROR(HttpServletResponse.SC_INTERNAL_SERVER_ERROR, "S3 server internal server error has occurred"); + + private final String errorCode; + private final int httpStatusCode; + private final String message; + + AzuErrorCode(int httpStatusCode, String message) { + this.errorCode = CaseFormat.UPPER_UNDERSCORE.to(CaseFormat.UPPER_CAMEL, + name()); + this.httpStatusCode = httpStatusCode; + this.message = requireNonNull(message); + } + + public String getErrorCode() { + return errorCode; + } + + public int getHttpStatusCode() { + return httpStatusCode; + } + + public String getMessage() { + return message; + } + + @Override + public String toString() { + return getHttpStatusCode() + " " + getErrorCode() + " " + getMessage(); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/exception/AzuException.java b/core/src/com/pspace/ifs/ksan/gw/exception/AzuException.java new file mode 100644 index 00000000..06bb5dc7 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/exception/AzuException.java @@ -0,0 +1,81 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.exception; + +import static java.util.Objects.requireNonNull; + +import java.util.Map; + +import com.google.common.collect.ImmutableMap; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; + +@SuppressWarnings("serial") +public final class AzuException extends Exception { + private final AzuErrorCode error; + private final Map elements; + private AzuParameter parameter = null; + + public AzuException(AzuErrorCode error) { + this(error, error.getMessage(), (Throwable) null, + ImmutableMap.of()); + } + + public AzuException(AzuErrorCode error, String message, Throwable cause, + Map elements) { + super(requireNonNull(message), cause); + this.error = requireNonNull(error); + this.elements = ImmutableMap.copyOf(elements); + } + + public AzuException(AzuErrorCode error, Throwable cause, AzuParameter parameter) { + this(error, error.getMessage(), cause, + ImmutableMap.of(), parameter); + } + + public AzuException(AzuErrorCode error, String message, AzuParameter parameter) { + this(error, message, (Throwable) null, + ImmutableMap.of(), parameter); + } + + public AzuException(AzuErrorCode error, String message, Throwable cause, Map elements, AzuParameter parameter) { + super(requireNonNull(message), cause); + this.parameter = parameter; + if(this.parameter != null) { + this.parameter.setErrorCode(error.getErrorCode()); + this.parameter.setStatusCode(error.getHttpStatusCode()); + } + + this.error = requireNonNull(error); + this.elements = ImmutableMap.copyOf(elements); + } + + public AzuException(AzuErrorCode error, AzuParameter parameter) { + this(error, error.getMessage(), (Throwable) null, ImmutableMap.of(), parameter); + } + + public AzuErrorCode getError() { + return error; + } + + public Map getElements() { + return elements; + } + + public AzuParameter getAZUParameter() { + return parameter; + } + + @Override + public String toString() { + return error + " " + elements; + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/exception/GWErrorCode.java b/core/src/com/pspace/ifs/ksan/gw/exception/GWErrorCode.java index 486a9ca9..5965e36f 100644 --- a/core/src/com/pspace/ifs/ksan/gw/exception/GWErrorCode.java +++ b/core/src/com/pspace/ifs/ksan/gw/exception/GWErrorCode.java @@ -58,6 +58,8 @@ public enum GWErrorCode { INVALID_REQUEST(HttpServletResponse.SC_BAD_REQUEST, "Bad Request"), INVALID_TAG(HttpServletResponse.SC_BAD_REQUEST, "The requested tagging key or value exceeds length"), + INVALID_TARGET_BUCKET_FOR_LOGGING(HttpServletResponse.SC_BAD_REQUEST, + "InvalidTargetBucketForLogging"), INVALID_EXCEED_TAG(HttpServletResponse.SC_BAD_REQUEST, "The requested tagging count has not exceed 10"), INVALID_CORS_ORIGIN(HttpServletResponse.SC_BAD_REQUEST, @@ -67,6 +69,7 @@ public enum GWErrorCode { MALFORMED_X_M_L(HttpServletResponse.SC_BAD_REQUEST, "The XML you provided was not well-formed or did not validate" + " against our published schema."), + MALFORMED_ACL(HttpServletResponse.SC_BAD_REQUEST, "MalformedACLError"), MAX_MESSAGE_LENGTH_EXCEEDED(HttpServletResponse.SC_BAD_REQUEST, "Your request was too big."), METHOD_NOT_ALLOWED(HttpServletResponse.SC_METHOD_NOT_ALLOWED, diff --git a/core/src/com/pspace/ifs/ksan/gw/format/AccessControlPolicy.java b/core/src/com/pspace/ifs/ksan/gw/format/AccessControlPolicy.java index e1319444..ec372e03 100644 --- a/core/src/com/pspace/ifs/ksan/gw/format/AccessControlPolicy.java +++ b/core/src/com/pspace/ifs/ksan/gw/format/AccessControlPolicy.java @@ -31,6 +31,8 @@ public String toString() { sb.append(GWConstants.LEFT_BRACE); if (owner != null) { sb.append(GWConstants.ACCESS_OW).append(owner); + } else { + sb.append(GWConstants.ACCESS_OW_EMPTY); } if (aclList != null) { diff --git a/core/common/common/ResponseDataWithModifiedBy.py b/core/src/com/pspace/ifs/ksan/gw/format/BlockList.java similarity index 59% rename from core/common/common/ResponseDataWithModifiedBy.py rename to core/src/com/pspace/ifs/ksan/gw/format/BlockList.java index 925359fc..c10a2a5c 100644 --- a/core/common/common/ResponseDataWithModifiedBy.py +++ b/core/src/com/pspace/ifs/ksan/gw/format/BlockList.java @@ -1,4 +1,4 @@ -""" +/* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version @@ -7,15 +7,21 @@ * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" +*/ +package com.pspace.ifs.ksan.gw.format; -from datetime import datetime -from ResponseDataWithCreatedBy import ResponseDataWithCreatedBy +import java.util.Collection; +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper; +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; + +// CHECKSTYLE:OFF +public final class BlockList { + @JacksonXmlProperty(localName = AzuConstants.LATEST) + @JacksonXmlElementWrapper(useWrapping = false) + public + Collection latests; +} +// CHECKSTYLE:ON -class ResponseDataWithModifiedBy(ResponseDataWithCreatedBy): - def __init__(self, regdate: datetime = None, regid: str = "", regname: str = "", moddate: datetime = None, modid: str = "", modname: str = ""): - super().__init__(regdate, regid, regname); - self.ModDate = (datetime.datetime.now() if moddate is None else regdate) - self.ModId = modid - self.ModName = modname diff --git a/portal/Portal/src/app/errors/400/error-400-page.component.ts b/core/src/com/pspace/ifs/ksan/gw/format/LegalHold.java similarity index 62% rename from portal/Portal/src/app/errors/400/error-400-page.component.ts rename to core/src/com/pspace/ifs/ksan/gw/format/LegalHold.java index a2e78b81..2ee67ddc 100644 --- a/portal/Portal/src/app/errors/400/error-400-page.component.ts +++ b/core/src/com/pspace/ifs/ksan/gw/format/LegalHold.java @@ -1,26 +1,25 @@ /* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version +* the GNU General Public License as published by the Free Software Foundation, either version * 3 of the License. See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ -import {Component, OnInit} from '@angular/core'; +package com.pspace.ifs.ksan.gw.format; -@Component({ - selector: 'app-error-400-page', - templateUrl: './error-400-page.component.html', - styleUrls: ['./error-400-page.component.scss'] -}) -export class Error400PageComponent implements OnInit { +import java.util.Collection; - constructor() { - } - - ngOnInit(): void { - } +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +/** Represent an Amazon Object Legal Hold. */ +// CHECKSTYLE:OFF +public final class LegalHold { + @JacksonXmlProperty(localName = GWConstants.XML_STATUS) + public + String status; } +// CHECKSTYLE:ON \ No newline at end of file diff --git a/core/src/com/pspace/ifs/ksan/gw/format/LoggingConfiguration.java b/core/src/com/pspace/ifs/ksan/gw/format/LoggingConfiguration.java new file mode 100644 index 00000000..a5034845 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/format/LoggingConfiguration.java @@ -0,0 +1,68 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.format; + +import java.util.Collection; + +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper; +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; +import com.pspace.ifs.ksan.gw.utils.GWConstants; + +/** Represent an Amazon Versioning for a container or object. */ +// CHECKSTYLE:OFF +public final class LoggingConfiguration { + @JacksonXmlElementWrapper(useWrapping=false) + @JacksonXmlProperty(localName = "LoggingEnabled") + public + LoggingEnabled loggingEnabled; + + public static final class LoggingEnabled { + @JacksonXmlProperty(localName = "TargetBucket") + public + String targetBucket; + + @JacksonXmlProperty(localName = "TargetPrefix") + public + String targetPrefix; + + @JacksonXmlProperty(localName = "TargetGrants") + public + TargetGrants targetGrants; + + public static final class TargetGrants { + @JacksonXmlProperty(localName = "Grant") + public + Collection grants; + + public static final class Grant { + @JacksonXmlProperty(localName = "Grantee") + public Grantee grantee; + @JacksonXmlProperty(localName = "Permission") + public String permission; + + public static final class Grantee { + @JacksonXmlProperty(namespace="http://www.w3.org/2001/XMLSchema-instance", localName="type", isAttribute = true) + public String type; + @JacksonXmlProperty(localName = "ID") + public String id; + @JacksonXmlProperty(localName = "DisplayName") + public String displayName; + @JacksonXmlProperty(localName = "EmailAddress") + public String emailAddress; + @JacksonXmlProperty(localName = "URI") + public String uri; + } + } + } + } +} +// CHECKSTYLE:ON + diff --git a/core/common/common/ResponseDataWithData.py b/core/src/com/pspace/ifs/ksan/gw/format/Retention.java similarity index 57% rename from core/common/common/ResponseDataWithData.py rename to core/src/com/pspace/ifs/ksan/gw/format/Retention.java index 1bfbb5ca..0be6913e 100644 --- a/core/common/common/ResponseDataWithData.py +++ b/core/src/com/pspace/ifs/ksan/gw/format/Retention.java @@ -1,4 +1,4 @@ -""" +/* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version @@ -7,22 +7,24 @@ * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" +*/ +package com.pspace.ifs.ksan.gw.format; -from typing import TypeVar, Generic, Type +import java.util.Collection; -from Enums.EnumResponseResult import EnumResponseResult +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper; +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; +import com.pspace.ifs.ksan.gw.utils.GWConstants; -T = TypeVar('T') +/** Represent an Amazon Retention. */ +// CHECKSTYLE:OFF +public class Retention { + @JacksonXmlProperty(localName = GWConstants.XML_MODE) + public + String mode; - -class ResponseDataWithData(Generic[T]): - def __init__(self, result: EnumResponseResult = EnumResponseResult.Error, code: str = "", message: str = "", value: T = None) -> None: - super().__init__(); - self.Result = result - self.Code = code - self.Message = message - self.Data = value - - def load(self, data: dict): - self.__dict__ = data.copy() + @JacksonXmlProperty(localName = GWConstants.RETAIN_UNTIL_DATE) + public + String date; +} +// CHECKSTYLE:ON diff --git a/core/src/com/pspace/ifs/ksan/gw/format/ServerSideEncryption.java b/core/src/com/pspace/ifs/ksan/gw/format/ServerSideEncryption.java index 8df67160..c601a24a 100644 --- a/core/src/com/pspace/ifs/ksan/gw/format/ServerSideEncryption.java +++ b/core/src/com/pspace/ifs/ksan/gw/format/ServerSideEncryption.java @@ -39,7 +39,7 @@ public static final class ApplyServerSideEncryptionByDefault { String KMSMasterKeyID; } - @JacksonXmlProperty(localName = GWConstants.OBJECT_LOCK_ENABLED) + @JacksonXmlProperty(localName = GWConstants.BUCKET_KEY_ENABLED) public String BucketKeyEnabled; } diff --git a/core/common/common/ResponseDataWithCreatedBy.py b/core/src/com/pspace/ifs/ksan/gw/format/TagIndex.java similarity index 65% rename from core/common/common/ResponseDataWithCreatedBy.py rename to core/src/com/pspace/ifs/ksan/gw/format/TagIndex.java index 1e1794a4..cce8486b 100644 --- a/core/common/common/ResponseDataWithCreatedBy.py +++ b/core/src/com/pspace/ifs/ksan/gw/format/TagIndex.java @@ -1,4 +1,4 @@ -""" +/* * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version @@ -7,12 +7,18 @@ * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -""" +*/ +package com.pspace.ifs.ksan.gw.format; -from datetime import datetime, date +import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; +import com.pspace.ifs.ksan.gw.utils.GWConstants; + +/** Represent an Amazon Versioning for a container or object. */ +// CHECKSTYLE:OFF +public final class TagIndex { + @JacksonXmlProperty(localName = GWConstants.XML_STATUS) + public + String status; +} +// CHECKSTYLE:ON -class ResponseDataWithCreatedBy: - def __init__(self, regdate: datetime = None, id: str = "", name: str = ""): - self.RegDate = (datetime.datetime.now() if regdate is None else regdate) - self.RegId = id - self.RegName = name diff --git a/core/src/com/pspace/ifs/ksan/gw/handler/Azu.java b/core/src/com/pspace/ifs/ksan/gw/handler/Azu.java new file mode 100644 index 00000000..ecb1a46a --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/handler/Azu.java @@ -0,0 +1,117 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.handler; + +import java.io.BufferedReader; +import java.io.File; +import java.io.FileReader; +import java.io.IOException; + +import org.eclipse.jetty.http.HttpCompliance; +import org.eclipse.jetty.http.UriCompliance; +import org.eclipse.jetty.server.HttpConfiguration; +import org.eclipse.jetty.server.HttpConnectionFactory; +import org.eclipse.jetty.server.ProxyConnectionFactory; +import org.eclipse.jetty.server.Server; +import org.eclipse.jetty.server.ServerConnector; +import org.eclipse.jetty.util.ssl.SslContextFactory; +import org.eclipse.jetty.util.thread.ExecutorThreadPool; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.identity.AzuSharedKey; +import com.pspace.ifs.ksan.gw.utils.AzuConfig; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.S3UserManager; +import com.pspace.ifs.ksan.gw.utils.S3UserManager; +import com.pspace.ifs.ksan.libs.PrintStack; + +public class Azu { + private static final Logger logger = LoggerFactory.getLogger(Azu.class); + private Server server; + private AzuHandlerJetty handler; + + public Azu() { + } + + public void configure() { + AzuConfig.getInstance().configure(); + } + + public void init() { + logger.info("azu start!"); + // configure(); + + ExecutorThreadPool pool = new ExecutorThreadPool(100); + pool.setName("AZU"); + server = new Server(pool); + + // The HTTP configuration object. + HttpConfiguration httpConfig = new HttpConfiguration(); + // Configure the HTTP support, for example: + httpConfig.setSendServerVersion(false); + + HttpConnectionFactory httpConnectionFactory = new HttpConnectionFactory(httpConfig); + HttpCompliance customHttpCompliance = HttpCompliance.from("RFC7230,MULTIPLE_CONTENT_LENGTHS"); + httpConnectionFactory.getHttpConfiguration().setHttpCompliance(customHttpCompliance); + UriCompliance customUriCompliance = UriCompliance.from("RFC3986,-AMBIGUOUS_PATH_SEPARATOR"); + httpConnectionFactory.getHttpConfiguration().setUriCompliance(customUriCompliance); + + ProxyConnectionFactory httpProxyConnectionFactory = new ProxyConnectionFactory(httpConnectionFactory.getProtocol()); + ServerConnector connector = new ServerConnector(server, httpProxyConnectionFactory, httpConnectionFactory); + connector.setHost("0.0.0.0"); + connector.setPort(10000); + + connector.setIdleTimeout(60000); + + connector.setReuseAddress(true); + server.addConnector(connector); + + handler = new AzuHandlerJetty(); + server.setHandler(handler); + + // shared key init + while (S3UserManager.getInstance().getUserSet().isEmpty()) { + logger.debug("wait s3user manager init ..."); + try { + Thread.sleep(1000); + } catch (Exception e) { + logger.error("", e); + } + } + + // File sharedKeys = new File(AzuConstants.SHARED_KEY_AUTH_PATH); + // if (sharedKeys.exists()) { + // try (BufferedReader reader = new BufferedReader(new FileReader(sharedKeys))) { + // String line = null; + // while ((line = reader.readLine()) != null) { + // String[] keyInfo = line.split(":"); + // AzuSharedKey key = new AzuSharedKey(keyInfo[0], keyInfo[1]); + // AzuSharedKeyManager.getInstance().addUser(key); + // } + // } catch (IOException e) { + // PrintStack.logging(logger, e); + // } + // } + } + + public void start() throws Exception { + server.start(); + } + + public void join() throws Exception { + server.join(); + } + + public void stop() throws Exception { + server.stop(); + } +} \ No newline at end of file diff --git a/core/src/com/pspace/ifs/ksan/gw/handler/AzuHandler.java b/core/src/com/pspace/ifs/ksan/gw/handler/AzuHandler.java new file mode 100644 index 00000000..5bc76989 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/handler/AzuHandler.java @@ -0,0 +1,297 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.handler; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.io.UnsupportedEncodingException; +import java.io.Writer; +import java.net.URLDecoder; +import java.text.DateFormat; +import java.text.SimpleDateFormat; +import java.util.Collections; +import java.util.Date; +import java.util.Map; +import java.util.TimeZone; +import java.util.UUID; +import java.util.regex.Matcher; + +import org.apache.http.HttpHeaders; +import org.eclipse.jetty.server.Request; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import jakarta.servlet.http.HttpServletRequest; +import jakarta.servlet.http.HttpServletResponse; + +import javax.xml.stream.XMLOutputFactory; +import javax.xml.stream.XMLStreamException; +import javax.xml.stream.XMLStreamWriter; + +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.api.azure.*; +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.sign.AzuSigning; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.libs.PrintStack; + +public class AzuHandler { + private static final Logger logger = LoggerFactory.getLogger(AzuHandler.class); + private final XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); + private AzuRequestFactory azuRequestFactory; + private String requestId; + + public AzuHandler() { + azuRequestFactory = new AzuRequestFactory(); + } + + public final void doHandle(Request baseRequest, HttpServletRequest request, HttpServletResponse response, InputStream is) throws AzuException { + long requestSize = 0L; + String method = request.getMethod().toUpperCase(); + requestSize = method.length(); + + String uri = request.getRequestURI(); + requestSize += uri.length(); + + long startTime = System.currentTimeMillis(); + + AzuParameter azuParameter = new AzuParameter(); + + logger.info(AzuConstants.LOG_PRE_URI, uri); + uri = removeDuplicateRoot(uri); + logger.info(AzuConstants.LOG_URI, uri); + + logger.info(AzuConstants.LOG_CLIENT_ADDRESS, request.getRemoteAddr()); + logger.info(AzuConstants.LOG_CLIENT_HOST, request.getRemoteHost()); + logger.info(AzuConstants.LOG_METHOD, method); + + for (String parameter : Collections.list(request.getParameterNames())) { + logger.info(AzuConstants.LOG_PARAMETER, parameter, Strings.nullToEmpty(request.getParameter(parameter))); + if (parameter.equalsIgnoreCase(AzuConstants.PARAMETER_COMP)) { + azuParameter.setComp(request.getParameter(parameter)); + } else if (parameter.equalsIgnoreCase(AzuConstants.PARAMETER_RESTYPE)) { + azuParameter.setRestype(request.getParameter(parameter)); + } else if (parameter.equalsIgnoreCase(AzuConstants.PARAMETER_PREFIX)) { + azuParameter.setPrefix(request.getParameter(parameter)); + } else if (parameter.equalsIgnoreCase(AzuConstants.PARAMETER_DELIMITER)) { + azuParameter.setDelimiter(request.getParameter(parameter)); + } else if (parameter.equalsIgnoreCase(AzuConstants.PARAMETER_INCLUDE)) { + azuParameter.setInclude(request.getParameter(parameter)); + } else if (parameter.equalsIgnoreCase(AzuConstants.PARAMETER_MAX_RESULTS)) { + azuParameter.setMaxresults(request.getParameter(parameter)); + } else if (parameter.equalsIgnoreCase(AzuConstants.PARAMETER_TIMEOUT)) { + azuParameter.setTimeout(request.getParameter(parameter)); + } else if (parameter.equalsIgnoreCase(AzuConstants.PARAMETER_MARKER)) { + azuParameter.setMarker(request.getParameter(parameter)); + } + requestSize += parameter.length(); + if (!Strings.isNullOrEmpty(request.getParameter(parameter))) { + requestSize += request.getParameter(parameter).length(); + } + } + + for (String headerName : Collections.list(request.getHeaderNames())) { + for (String headerValue : Collections.list(request.getHeaders(headerName))) { + logger.info(AzuConstants.LOG_HEADER, headerName, Strings.nullToEmpty(headerValue)); + if (headerName.equalsIgnoreCase(AzuConstants.X_MS_CLIENT_REQUEST_ID)) { + requestId = Strings.nullToEmpty(headerValue); + } + + requestSize += headerName.length(); + if (!Strings.isNullOrEmpty(headerValue)) { + requestSize += headerValue.length(); + } + } + } + + // make request id + String requestID = UUID.randomUUID().toString().substring(24).toUpperCase(); + String userName = ""; + String containerName = ""; + String blobName = ""; + String pathCategory = ""; + + try { + String[] path = uri.split(AzuConstants.SEPARATOR, 4); + try { + for (int i = 0; i < path.length; i++) { + path[i] = URLDecoder.decode(path[i], AzuConstants.CHARSET_UTF_8); + logger.info(AzuConstants.LOG_PATH, i, path[i]); + } + } catch (UnsupportedEncodingException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.BAD_REQUEST, null); + } + + userName = path[1]; + if (path.length > 2) { + containerName = path[2]; + // $logs, $blobchangfeed + if (containerName.equalsIgnoreCase(AzuConstants.REQUEST_LOGS) || containerName.equalsIgnoreCase(AzuConstants.REQUEST_BLOCK_CHANGE_FEED)) { + logger.info(AzuConstants.LOG_LOGS_BLOB_CHANGE_FEED, containerName); + response.setStatus(HttpServletResponse.SC_NOT_FOUND); + return; + } + } + if (path.length > 3) { + blobName = path[3]; + if (!File.separator.equals(AzuConstants.SEPARATOR)) { + blobName = blobName.replaceAll(AzuConstants.SEPARATOR, Matcher.quoteReplacement(File.separator)); + } + } + + if (path.length == 2) { + pathCategory = AzuConstants.PATH_CATEGORY_ROOT; + } else if (path.length == 3) { + pathCategory = AzuConstants.PATH_CATEGORY_CONTAINER; + } else if (path.length == 4) { + pathCategory = AzuConstants.PATH_CATEGORY_BLOB; + } + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, null); + } + + azuParameter.setMethod(method); + azuParameter.setRequest(request); + azuParameter.setResponse(response); + azuParameter.setInputStream(is); + azuParameter.setUserName(userName); + azuParameter.setContainerName(containerName); + azuParameter.setBlobName(blobName); + azuParameter.setPathCategory(pathCategory); + + try { + azuParameter.setUser(new AzuSigning(azuParameter).getUser()); + } catch (Exception e) { + writeSigningError(request, response); + } + + if (azuParameter.getComp().equalsIgnoreCase(AzuConstants.COMP_PROPERTIES) && azuParameter.getRestype().equalsIgnoreCase(AzuConstants.RESTYPE_ACCOUNT)) { + logger.info("request : account, send SC_OK"); + response.setStatus(HttpServletResponse.SC_OK); + return; + } + + AzuRequest azuRequest = azuRequestFactory.createRequest(azuParameter); + azuRequest.process(); + } + + private String removeDuplicateRoot(String s) { + boolean find = false; + int sLength = s.length(); + StringBuilder resultStr = new StringBuilder(); + for (int i = 0; i < sLength; i++) { + char tempChar = s.charAt(i); + int tempVal = (int) tempChar; + + if (tempVal == AzuConstants.SEPARATOR_CHAR) { + if (find == true) { + continue; + } + + find = true; + } else { + find = false; + } + + resultStr.append(tempChar); + } + return resultStr.toString(); + } + + private void writeSigningError(HttpServletRequest request, HttpServletResponse response) { + response.setStatus(HttpServletResponse.SC_FORBIDDEN, "Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature."); + response.setCharacterEncoding(AzuConstants.CHARSET_UTF_8); + try { + XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); + try (Writer writer = response.getWriter()) { + response.setContentType(AzuConstants.CONTENT_TYPE_XML); + XMLStreamWriter xmlStreamWriter = xmlOutputFactory.createXMLStreamWriter(writer); + xmlStreamWriter.writeStartDocument(); + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_ERROR); + + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_CODE); + xmlStreamWriter.writeCharacters("AuthorizationFailure"); + xmlStreamWriter.writeEndElement(); // End Code + xmlStreamWriter.writeStartElement(AzuConstants.XML_ELEMENT_MESSAGE); + String message = "Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.\n" + "RequestId:" + request.getHeader(AzuConstants.X_MS_CLIENT_REQUEST_ID) + "\nTime:"; + TimeZone tz = TimeZone.getTimeZone(GWConstants.UTC); + DateFormat df = new SimpleDateFormat(AzuConstants.TIME_FORMAT); + df.setTimeZone(tz); + String nowTime = df.format(new Date()); + message += nowTime; + xmlStreamWriter.writeEndElement(); // End Message + xmlStreamWriter.writeEndElement(); // End Error + xmlStreamWriter.flush(); + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } + + protected final void sendSimpleErrorResponse( + HttpServletRequest request, HttpServletResponse response, + AzuErrorCode code, String message, + Map elements, AzuParameter parameter) throws IOException { + + response.setStatus(code.getHttpStatusCode()); + + if (request.getMethod().equals("HEAD")) { + // The HEAD method is identical to GET except that the server MUST + // NOT return a message-body in the response. + return; + } + + try (Writer writer = response.getWriter()) { + response.setContentType(AzuConstants.CONTENT_TYPE_XML); + XMLStreamWriter xml = xmlOutputFactory.createXMLStreamWriter(writer); + xml.writeStartDocument(); + xml.writeStartElement(AzuConstants.XML_ELEMENT_ERROR); + + writeSimpleElement(xml, AzuConstants.XML_ELEMENT_CODE, "400"); + writeSimpleElement(xml, AzuConstants.XML_ELEMENT_MESSAGE, message); + + // for (Map.Entry entry : elements.entrySet()) { + // writeSimpleElement(xml, entry.getKey(), entry.getValue()); + // } + + writeSimpleElement(xml, AzuConstants.REQUEST_ID, requestId); + + xml.writeEndElement(); + xml.flush(); + } catch (XMLStreamException xse) { + throw new IOException(xse); + } + } + + private void sendException(HttpServletRequest request, HttpServletResponse response, AzuException se) throws IOException { + sendSimpleErrorResponse( + request, + response, + se.getError(), + se.getMessage(), + se.getElements(), + se.getAZUParameter()); + } + + private void writeSimpleElement(XMLStreamWriter xml, String elementName, String characters) throws XMLStreamException { + xml.writeStartElement(elementName); + xml.writeCharacters(characters); + xml.writeEndElement(); + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/handler/AzuHandlerJetty.java b/core/src/com/pspace/ifs/ksan/gw/handler/AzuHandlerJetty.java new file mode 100644 index 00000000..7977efe7 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/handler/AzuHandlerJetty.java @@ -0,0 +1,58 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.handler; + +import java.io.IOException; +import java.io.InputStream; + +import org.apache.http.HttpHeaders; +import org.eclipse.jetty.server.Request; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.eclipse.jetty.server.Request; +import org.eclipse.jetty.server.handler.AbstractHandler; + +import jakarta.servlet.ServletException; +import jakarta.servlet.http.HttpServletRequest; +import jakarta.servlet.http.HttpServletResponse; + +import com.pspace.ifs.ksan.gw.exception.AzuException; + +public class AzuHandlerJetty extends AbstractHandler { + private static final Logger logger = LoggerFactory.getLogger(AzuHandlerJetty.class); + private final AzuHandler handler; + + public AzuHandlerJetty() { + handler = new AzuHandler(); + } + + private void sendAZUException(HttpServletRequest request, HttpServletResponse response, AzuException e) + throws IOException { + handler.sendSimpleErrorResponse(request, response, e.getError(), e.getMessage(), e.getElements(), e.getAZUParameter()); + } + + @Override + public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) + throws IOException, ServletException { + try (InputStream is = request.getInputStream()) { + handler.doHandle(baseRequest, request, response, is); + baseRequest.setHandled(true); + } catch (AzuException e) { + sendAZUException(request, response, e); + baseRequest.setHandled(true); + } + } + + public AzuHandler getHandler() { + return handler; + } +} \ No newline at end of file diff --git a/core/src/com/pspace/ifs/ksan/gw/handler/GW.java b/core/src/com/pspace/ifs/ksan/gw/handler/GW.java index b9c48767..3601e6bc 100644 --- a/core/src/com/pspace/ifs/ksan/gw/handler/GW.java +++ b/core/src/com/pspace/ifs/ksan/gw/handler/GW.java @@ -16,6 +16,7 @@ import com.pspace.ifs.ksan.gw.db.GWDB; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.object.objmanager.ObjManagerHelper; +import com.pspace.ifs.ksan.gw.object.objmanager.ObjManagers; import com.pspace.ifs.ksan.gw.object.osdclient.OSDClientManager; import com.pspace.ifs.ksan.gw.utils.GWConfig; import com.pspace.ifs.ksan.gw.utils.GWConstants; @@ -108,9 +109,9 @@ public void init() throws Exception { connector.setHost(GWConfig.getInstance().getEndpoint().getHost()); connector.setPort(GWConfig.getInstance().getEndpoint().getPort()); - // if(GWConfig2.getInstance().jettyMaxIdleTimeout() > 30000) { + if (GWConfig.getInstance().getJettyMaxIdleTimeout() > 30000) { connector.setIdleTimeout(GWConfig.getInstance().getJettyMaxIdleTimeout()); - // } + } connector.setReuseAddress(true); server.addConnector(connector); @@ -125,7 +126,7 @@ public void init() throws Exception { connector = new ServerConnector(server, sslContextFactory, httpConnectionFactory); connector.setHost(GWConfig.getInstance().getSecureEndpoint().getHost()); connector.setPort(GWConfig.getInstance().getSecureEndpoint().getPort()); - if(GWConfig.getInstance().getJettyMaxIdleTimeout() > 30000) { + if (GWConfig.getInstance().getJettyMaxIdleTimeout() > 30000) { connector.setIdleTimeout(GWConfig.getInstance().getJettyMaxIdleTimeout()); } @@ -138,24 +139,25 @@ public void init() throws Exception { handler = new GWHandlerJetty(); server.setHandler(handler); - GWDB s3DB = GWUtils.getDBInstance(); - try { - s3DB.init(GWConfig.getInstance().getDbHost(), String.valueOf(GWConfig.getInstance().getDbPort()), GWConfig.getInstance().getDatabase(), GWConfig.getInstance().getDbUser(), GWConfig.getInstance().getDbPass(), (int)GWConfig.getInstance().getDbPoolSize()); - } catch (Exception e) { - PrintStack.logging(logger, e); - } + // GWDB s3DB = GWUtils.getDBInstance(); + // try { + // s3DB.init(GWConfig.getInstance().getDbHost(), String.valueOf(GWConfig.getInstance().getDbPort()), GWConfig.getInstance().getDatabase(), GWConfig.getInstance().getDbUser(), GWConfig.getInstance().getDbPass(), (int)GWConfig.getInstance().getDbPoolSize()); + // } catch (Exception e) { + // PrintStack.logging(logger, e); + // } - try { - OSDClientManager.getInstance().init((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount()); - } catch (Exception e) { - PrintStack.logging(logger, e); - } + // try { + // OSDClientManager.getInstance().init((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount()); + // } catch (Exception e) { + // PrintStack.logging(logger, e); + // } try { ObjManagerHelper.getInstance().init((int)GWConfig.getInstance().getObjManagerCount()); } catch (Exception e) { PrintStack.logging(logger, e); } + // ObjManagers.getInstance().init(); if (GWConfig.getInstance().isCacheDiskpath()) { GWUtils.initCache(GWConfig.getInstance().getCacheDiskpath()); diff --git a/core/src/com/pspace/ifs/ksan/gw/handler/GWHandler.java b/core/src/com/pspace/ifs/ksan/gw/handler/GWHandler.java index 81219668..59909614 100644 --- a/core/src/com/pspace/ifs/ksan/gw/handler/GWHandler.java +++ b/core/src/com/pspace/ifs/ksan/gw/handler/GWHandler.java @@ -31,14 +31,16 @@ import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.gw.object.objmanager.ObjManagerHelper; +import com.pspace.ifs.ksan.gw.object.objmanager.ObjManagers; import com.pspace.ifs.ksan.gw.object.osdclient.OSDClientManager; import com.pspace.ifs.ksan.gw.sign.S3Signing; import com.pspace.ifs.ksan.gw.utils.AsyncHandler; import com.pspace.ifs.ksan.gw.utils.GWConfig; -import com.pspace.ifs.ksan.gw.utils.GWConfig; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; +import com.pspace.ifs.ksan.objmanager.ObjManager; import org.apache.http.HttpHeaders; import org.eclipse.jetty.server.Request; @@ -68,6 +70,23 @@ public GWHandler() { } public final void doHandle(Request baseRequest, HttpServletRequest request, HttpServletResponse response, InputStream is) throws GWException { + if (GWConfig.getInstance().isNoOperation()) { + response.setStatus(HttpServletResponse.SC_OK); + return; + } + + // if (GWConfig.getInstance().isDBOP()) { + // ObjManager objManager = ObjManagers.getInstance().getObjManager(); + // try { + // objManager.open("db-test", "file1.txt"); + // } catch (Exception e) { + // logger.error("Failed to open"); + // } + + // response.setStatus(HttpServletResponse.SC_OK); + // return; + // } + long requestSize = 0L; String method = request.getMethod(); requestSize = method.length(); @@ -87,7 +106,6 @@ public final void doHandle(Request baseRequest, HttpServletRequest request, Http for (String parameter : Collections.list(request.getParameterNames())) { logger.info(GWConstants.LOG_GWHANDLER_PARAMETER, parameter, Strings.nullToEmpty(request.getParameter(parameter))); - requestSize += parameter.length(); if (!Strings.isNullOrEmpty(request.getParameter(parameter))) { requestSize += request.getParameter(parameter).length(); @@ -97,7 +115,6 @@ public final void doHandle(Request baseRequest, HttpServletRequest request, Http for (String headerName : Collections.list(request.getHeaderNames())) { for (String headerValue : Collections.list(request.getHeaders(headerName))) { logger.info(GWConstants.LOG_GWHANDLER_HEADER, headerName, Strings.nullToEmpty(headerValue)); - requestSize += headerName.length(); if (!Strings.isNullOrEmpty(headerValue)) { requestSize += headerValue.length(); @@ -111,7 +128,7 @@ public final void doHandle(Request baseRequest, HttpServletRequest request, Http String[] path = uri.split(GWConstants.SLASH, 3); try { for (int i = 0; i < path.length; i++) { - path[i] = URLDecoder.decode(path[i], GWConstants.CHARSET_UTF_8); + path[i] = URLDecoder.decode(path[i], Constants.CHARSET_UTF_8); logger.info(GWConstants.LOG_GWHANDLER_PATH, i, path[i]); } } catch (UnsupportedEncodingException e) { @@ -128,6 +145,9 @@ public final void doHandle(Request baseRequest, HttpServletRequest request, Http pathCategory = GWConstants.CATEGORY_OBJECT; } + // long infoRequest = System.currentTimeMillis(); + // logger.error("info time : {}", infoRequest - startTime); + S3Parameter s3Parameter = new S3Parameter(); s3Parameter.setURI(uri); s3Parameter.setRequestSize(requestSize); @@ -156,6 +176,8 @@ public final void doHandle(Request baseRequest, HttpServletRequest request, Http s3Parameter.setHostID(request.getHeader(GWConstants.X_AMZ_ID_2)); s3Parameter.setRemoteAddr(!Strings.isNullOrEmpty(request.getHeader(GWConstants.X_FORWARDED_FOR)) ? request.getHeader(GWConstants.X_FORWARDED_FOR) : request.getRemoteAddr()); + // long preSign = System.currentTimeMillis(); + // logger.error("para time : {}", preSign - startTime); S3Signing s3signing = new S3Signing(s3Parameter); if (request.getHeader(HttpHeaders.AUTHORIZATION) == null && request.getParameter(GWConstants.X_AMZ_ALGORITHM) == null @@ -177,6 +199,9 @@ public final void doHandle(Request baseRequest, HttpServletRequest request, Http s3Parameter.setAdmin(false); } + // long preTime = System.currentTimeMillis(); + // logger.error("sign time : {}", preTime - preSign); + S3Request s3Request = null; if (s3Parameter.isAdmin()) { logger.info(GWConstants.LOG_GWHANDLER_ADMIN_MOTHOD_CATEGORY, s3Parameter.getMethod(), s3Parameter.getPathCategory()); @@ -186,9 +211,17 @@ public final void doHandle(Request baseRequest, HttpServletRequest request, Http s3Request = s3RequestFactory.createS3Request(s3Parameter); + if (GWConfig.getInstance().isPREV()) { + response.setStatus(HttpServletResponse.SC_OK); + return; + } + s3Request.process(); s3Parameter.setStatusCode(response.getStatus()); AsyncHandler.s3logging(s3Parameter); + // long endTime = System.currentTimeMillis(); + // logger.error("rest time : {}", endTime - preTime); + // logger.error("work time : {}", endTime - startTime); } private String removeDuplicateRoot(String s) { diff --git a/core/src/com/pspace/ifs/ksan/gw/handler/GWHandlerJetty.java b/core/src/com/pspace/ifs/ksan/gw/handler/GWHandlerJetty.java index 14b3192e..babc3f2d 100644 --- a/core/src/com/pspace/ifs/ksan/gw/handler/GWHandlerJetty.java +++ b/core/src/com/pspace/ifs/ksan/gw/handler/GWHandlerJetty.java @@ -51,24 +51,6 @@ public void handle(String target, Request baseRequest, HttpServletRequest reques throws IOException, ServletException { // TODO Auto-generated method stub try (InputStream is = request.getInputStream()) { - // logger.info(baseRequest.getRootURL() + baseRequest.getOriginalURI()); - // List params = URLEncodedUtils.parse(baseRequest.getHttpURI().toURI(), Charset.forName(GWConstants.CHARSET_UTF_8)); - - // MultiMap queryParameters = new MultiMap(); - // for (NameValuePair param : params) { - // logger.info(param.getName() + GWConstants.SPACE_COLON_SPACE + param.getValue()); - - // String encodevalue = GWConstants.EMPTY_STRING; - // if(param.getValue() != null) { - // if(param.getName().equals(GWConstants.SIGNATURE)) - // encodevalue = param.getValue().replaceAll(GWConstants.SPACE, GWConstants.PLUS); - // else - // encodevalue = param.getValue(); - // } - - // queryParameters.put(param.getName(), encodevalue); - // } - // baseRequest.setQueryParameters(queryParameters); handler.doHandle(baseRequest, request, response, is); baseRequest.setHandled(true); } catch (GWException e) { diff --git a/core/src/com/pspace/ifs/ksan/gw/identity/AzuParameter.java b/core/src/com/pspace/ifs/ksan/gw/identity/AzuParameter.java new file mode 100644 index 00000000..02c27d07 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/identity/AzuParameter.java @@ -0,0 +1,207 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.identity; + +import java.io.InputStream; + +import jakarta.servlet.http.HttpServletRequest; +import jakarta.servlet.http.HttpServletResponse; + +import com.google.common.base.Strings; + +public class AzuParameter { + private HttpServletRequest request; + private HttpServletResponse response; + private InputStream inputStream; + private S3User user; + private String comp; + private String restype; + private String prefix; + private String delimiter; + private String marker; + private String include; + private String maxresults; + private String timeout; + + private String userName; + private String containerName; + private String blobName; + private String pathCategory; + private String method; + private int statusCode; + private String errorCode; + private long requestSize; + private long responseSize; + + public AzuParameter() { + request = null; + response = null; + inputStream = null; + user = null; + comp = ""; + restype = ""; + prefix = ""; + delimiter = ""; + marker = ""; + include = ""; + maxresults = ""; + timeout = ""; + userName = ""; + containerName = ""; + blobName = ""; + method = ""; + statusCode = 0; + errorCode = ""; + requestSize = 0L; + responseSize = 0L; + } + + public HttpServletRequest getRequest() { + return request; + } + public void setRequest(HttpServletRequest request) { + this.request = request; + } + public HttpServletResponse getResponse() { + return response; + } + public void setResponse(HttpServletResponse response) { + this.response = response; + } + public InputStream getInputStream() { + return inputStream; + } + public void setInputStream(InputStream inputStream) { + this.inputStream = inputStream; + } + public S3User getUser() { + return user; + } + + public void setUser(S3User user) { + this.user = user; + } + public String getComp() { + return comp; + } + public void setComp(String comp) { + this.comp = comp; + } + public String getRestype() { + return restype; + } + public void setRestype(String restype) { + this.restype = restype; + } + public String getPrefix() { + return prefix; + } + public void setDelimiter(String delimiter) { + this.delimiter = delimiter; + } + public String getDelimiter() { + return delimiter; + } + public void setMarker(String marker) { + this.marker = marker; + } + public String getMarker() { + return marker; + } + public void setInclude(String include) { + this.include = include; + } + public String getInclude() { + return include; + } + public void setMaxresults(String maxresults) { + this.maxresults = maxresults; + } + public String getMaxResults() { + return maxresults; + } + public void setTimeout(String timeout) { + this.timeout = timeout; + } + public String getTimeout() { + return timeout; + } + public void setPrefix(String prefix) { + this.prefix = prefix; + } + public void setUserName(String userName) { + this.userName = userName; + } + public String getUserName() { + return this.userName; + } + public String getContainerName() { + return containerName; + } + public void setContainerName(String containerName) { + this.containerName = containerName; + } + public String getBlobName() { + return blobName; + } + public void setBlobName(String blobName) { + this.blobName = blobName; + } + public String getPathCategory() { + return pathCategory; + } + public void setPathCategory(String pathCategory) { + this.pathCategory = pathCategory; + } + public String getMethod() { + return method; + } + public void setMethod(String method) { + this.method = method; + } + public int getStatusCode() { + return statusCode; + } + public void setStatusCode(int statusCode) { + this.statusCode = statusCode; + } + public String getErrorCode() { + return errorCode; + } + public void setErrorCode(String errorCode) { + this.errorCode = errorCode; + } + public void setRequestSize(long requestSize) { + this.requestSize = requestSize; + } + + public void addRequestSize(long size) { + this.requestSize += size; + } + + public long getRequestSize() { + return requestSize; + } + + public void addResponseSize(long size) { + this.responseSize += size; + } + + public void setResponseSize(long size) { + this.responseSize = size; + } + + public long getResponseSize() { + return responseSize; + } + +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/identity/AzuSharedKey.java b/core/src/com/pspace/ifs/ksan/gw/identity/AzuSharedKey.java new file mode 100644 index 00000000..f6ef5cc2 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/identity/AzuSharedKey.java @@ -0,0 +1,43 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.identity; + +public class AzuSharedKey { + private String userId; + private String userKey; + + public AzuSharedKey() { + userId = ""; + userKey = ""; + } + + public AzuSharedKey(String userId, String userKey) { + this.userId = userId; + this.userKey = userKey; + } + + public void setUserId(String userId) { + this.userId = userId; + } + + public String getUserId() { + return userId; + } + + public void setUserKey(String userKey) { + this.userKey = userKey; + } + + public String getUserKey() { + return userKey; + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/identity/ObjectListParameter.java b/core/src/com/pspace/ifs/ksan/gw/identity/ObjectListParameter.java deleted file mode 100644 index 204216ea..00000000 --- a/core/src/com/pspace/ifs/ksan/gw/identity/ObjectListParameter.java +++ /dev/null @@ -1,94 +0,0 @@ -/* -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -*/ -package com.pspace.ifs.ksan.gw.identity; - -import java.util.ArrayList; -import java.util.List; -import java.util.SortedMap; -import java.util.TreeMap; - -import com.google.common.base.Strings; - -public class ObjectListParameter { - private boolean istruncated; - private String nextMarker; - private String nextVersion; - private String nextUploadid; - private int nextPartNumber; - private SortedMap commonPrefixes; - private List objects; - - public ObjectListParameter() { - istruncated = false; - nextMarker = ""; - nextVersion = ""; - nextUploadid = ""; - nextPartNumber = 0; - objects = new ArrayList(); - commonPrefixes = new TreeMap(); - } - - public boolean isTruncated() { - return istruncated; - } - - public void setIstruncated(boolean istruncated) { - this.istruncated = istruncated; - } - - public String getNextMarker() { - return Strings.nullToEmpty(nextMarker); - } - - public void setNextMarker(String nextMarker) { - this.nextMarker = nextMarker; - } - - public String getNextVersion() { - return Strings.nullToEmpty(nextVersion); - } - - public void setNextVersion(String nextVersion) { - this.nextVersion = nextVersion; - } - - public String getNextUploadid() { - return Strings.nullToEmpty(nextUploadid); - } - - public void setNextUploadid(String nextUploadid) { - this.nextUploadid = nextUploadid; - } - - public int getNextPartNumber() { - return nextPartNumber; - } - - public void setNextPartNumber(int nextPartNumber) { - this.nextPartNumber = nextPartNumber; - } - - public SortedMap getCommonPrefixes() { - return commonPrefixes; - } - - public void setCommonPrefixes(SortedMap commonPrefixes) { - this.commonPrefixes = commonPrefixes; - } - - public List getObjects() { - return objects; - } - - public void setObjects(List objects) { - this.objects = objects; - } -} diff --git a/core/src/com/pspace/ifs/ksan/gw/identity/S3Bucket.java b/core/src/com/pspace/ifs/ksan/gw/identity/S3Bucket.java index e72102c4..fd3b1bdd 100644 --- a/core/src/com/pspace/ifs/ksan/gw/identity/S3Bucket.java +++ b/core/src/com/pspace/ifs/ksan/gw/identity/S3Bucket.java @@ -27,6 +27,10 @@ public class S3Bucket { private String tagging; private String encryption; private String replication; + private String logging; + private String notification; + private String policy; + private String objectLock; private Date timestamp; public S3Bucket() { @@ -42,6 +46,10 @@ public S3Bucket() { tagging = ""; encryption = ""; replication = ""; + logging = ""; + notification = ""; + policy = ""; + objectLock = ""; timestamp = null; } @@ -123,4 +131,37 @@ public Date getTimestamp() { public void setTimestamp(Date timestamp) { this.timestamp = timestamp; } + + public String getLogging() { + return logging; + } + + public void setLogging(String logging) { + this.logging = logging; + } + + public String getNotification() { + return notification; + } + + public void setNotification(String notification) { + this.notification = notification; + } + + public String getPolicy() { + return policy; + } + + public void setPolicy(String policy) { + this.policy = policy; + } + + public String getObjectLock() { + return objectLock; + } + + public void setObjectLock(String objectLock) { + this.objectLock = objectLock; + } + } diff --git a/core/src/com/pspace/ifs/ksan/gw/identity/S3Metadata.java b/core/src/com/pspace/ifs/ksan/gw/identity/S3Metadata.java deleted file mode 100644 index 09c8d526..00000000 --- a/core/src/com/pspace/ifs/ksan/gw/identity/S3Metadata.java +++ /dev/null @@ -1,400 +0,0 @@ -/* -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -*/ -package com.pspace.ifs.ksan.gw.identity; - -import java.util.Date; -import java.util.HashMap; -import java.util.Map; - -import com.google.common.base.Strings; -import com.pspace.ifs.ksan.gw.utils.GWConstants; - -public class S3Metadata { - private String storageType; - private String name; - private String uri; - private Map userMetadata; - private String eTag; - private Date creationDate; - private Date lastModified; - private Date lastAccess; - private Long size; - private String tier; - private String cacheControl; - private String readEncryption; - private String readEncryptionSize; - private String customerAlgorithm; - private String customerKey; - private String customerKeyMD5; - private String serversideEncryption; - private String deleteMarker; - private String isLatest; - private Long contentLength; - private String contentDisposition; - private String contentEncoding; - private String contentType; - private String ownerId; - private String ownerName; - private String contentLanguage; - private Date expires; - private String taggingCount; - private long decodedContentLength; - private String contentMD5; - private String encryption; - private String versionId; - private String uploadId; - private int partNumber; - - private String lockMode; - private String lockExpires; - private String legalHold; - - public S3Metadata() { - storageType = ""; - name = ""; - uri = ""; - userMetadata = new HashMap(); - eTag = ""; - creationDate = null; - lastModified = null; - lastAccess = null; - size = 0L; - tier = GWConstants.AWS_TIER_STANTARD; - cacheControl = ""; - readEncryption = ""; - readEncryptionSize = ""; - customerAlgorithm = ""; - customerKey = ""; - customerKeyMD5 = ""; - serversideEncryption = ""; - deleteMarker = ""; - isLatest = ""; - contentLength = 0L; - contentDisposition = ""; - contentEncoding = ""; - contentType = ""; - ownerId = ""; - ownerName = ""; - contentLanguage = ""; - expires = null; - taggingCount = ""; - decodedContentLength = 0L; - contentMD5 = ""; - encryption = ""; - versionId = ""; - uploadId = ""; - partNumber = 0; - } - - public Map getUserMetadataMap() { - return userMetadata; - } - - public void setUserMetadataMap(Map userMetadata) { - this.userMetadata = userMetadata; - } - - public void addUserMetadata(String Key, String Value) { - userMetadata.put(Key, Value); - } - - public String getUserMetadata(String key) { - return userMetadata.get(key); - } - - public String getStorageType() { - return Strings.nullToEmpty(storageType); - } - - public void setStorageType(String storageType) { - this.storageType = storageType; - } - - public String getName() { - return Strings.nullToEmpty(name); - } - - public void setName(String name) { - this.name = name; - } - - public String getUri() { - return Strings.nullToEmpty(uri); - } - - public void setUri(String uri) { - this.uri = uri; - } - - public String getETag() { - return Strings.nullToEmpty(eTag); - } - - public void setETag(String eTag) { - this.eTag = eTag; - } - - public Date getCreationDate() { - return creationDate; - } - - public void setCreationDate(Date creationDate) { - this.creationDate = creationDate; - } - - public Date getLastModified() { - return lastModified; - } - - public void setLastModified(Date lastModified) { - this.lastModified = lastModified; - } - - public Date getLastAccess() { - return lastAccess; - } - - public void setLastAccess(Date lastAccess) { - this.lastAccess = lastAccess; - } - - public Long getSize() { - return size; - } - - public void setSize(Long size) { - this.size = size; - } - - public String getTier() { - return Strings.nullToEmpty(tier); - } - - public void setTier(String tier) { - this.tier = tier; - } - - public String getCacheControl() { - return Strings.nullToEmpty(cacheControl); - } - - public void setCacheControl(String cacheControl) { - this.cacheControl = cacheControl; - } - - public String getReadEncryption() { - return Strings.nullToEmpty(readEncryption); - } - - public void setReadEncryption(String readEncryption) { - this.readEncryption = readEncryption; - } - - public String getReadEncryptionSize() { - return Strings.nullToEmpty(readEncryptionSize); - } - - public void setReadEncryptionSize(String readEncryptionSize) { - this.readEncryptionSize = readEncryptionSize; - } - - public String getCustomerAlgorithm() { - return Strings.nullToEmpty(customerAlgorithm); - } - - public void setCustomerAlgorithm(String customerAlgorithm) { - this.customerAlgorithm = customerAlgorithm; - } - - public String getCustomerKey() { - return Strings.nullToEmpty(customerKey); - } - - public void setCustomerKey(String customerKey) { - this.customerKey = customerKey; - } - - public String getCustomerKeyMD5() { - return Strings.nullToEmpty(customerKeyMD5); - } - - public void setCustomerKeyMD5(String customerKeyMD5) { - this.customerKeyMD5 = customerKeyMD5; - } - - public String getServersideEncryption() { - return Strings.nullToEmpty(serversideEncryption); - } - - public void setServersideEncryption(String serversideEncryption) { - this.serversideEncryption = serversideEncryption; - } - - public String getDeleteMarker() { - return Strings.nullToEmpty(deleteMarker); - } - - public void setDeleteMarker(String deleteMarker) { - this.deleteMarker = deleteMarker; - } - - public String getIsLatest() { - return Strings.nullToEmpty(isLatest); - } - - public void setIsLatest(String isLatest) { - this.isLatest = isLatest; - } - - public Long getContentLength() { - return contentLength; - } - - public void setContentLength(Long contentLength) { - this.contentLength = contentLength; - } - - public String getContentDisposition() { - return Strings.nullToEmpty(contentDisposition); - } - - public void setContentDisposition(String contentDisposition) { - this.contentDisposition = contentDisposition; - } - - public String getContentEncoding() { - return Strings.nullToEmpty(contentEncoding); - } - - public void setContentEncoding(String contentEncoding) { - this.contentEncoding = contentEncoding; - } - - public String getContentType() { - return Strings.nullToEmpty(contentType); - } - - public void setContentType(String contentType) { - this.contentType = contentType; - } - - public String getOwnerId() { - return Strings.nullToEmpty(ownerId); - } - - public void setOwnerId(String ownerId) { - this.ownerId = ownerId; - } - - public String getOwnerName() { - return Strings.nullToEmpty(ownerName); - } - - public void setOwnerName(String ownerName) { - this.ownerName = ownerName; - } - - public String getContentLanguage() { - return Strings.nullToEmpty(contentLanguage); - } - - public void setContentLanguage(String contentLanguage) { - this.contentLanguage = contentLanguage; - } - - public Date getExpires() { - return expires; - } - - public void setExpires(Date expires) { - this.expires = expires; - } - - public String getTaggingCount() { - return Strings.nullToEmpty(taggingCount); - } - - public void setTaggingCount(String taggingCount) { - this.taggingCount = taggingCount; - } - - public long getDecodedContentLength() { - return decodedContentLength; - } - - public void setDecodedContentLength(long decodedContentLength) { - this.decodedContentLength = decodedContentLength; - } - - public String getContentMD5() { - return Strings.nullToEmpty(contentMD5); - } - - public void setContentMD5(String contentMD5) { - this.contentMD5 = contentMD5; - } - - public String getEncryption() { - return Strings.nullToEmpty(encryption); - } - - public void setEncryption(String encryption) { - this.encryption = encryption; - } - - public String getVersionId() { - return Strings.nullToEmpty(versionId); - } - - public void setVersionId(String versionId) { - this.versionId = versionId; - } - - public String getUploadId() { - return Strings.nullToEmpty(uploadId); - } - - public void setUploadId(String uploadId) { - this.uploadId = uploadId; - } - - public int getPartNumber() { - return partNumber; - } - - public void setPartNumber(int partNumber) { - this.partNumber = partNumber; - } - - public String getLockMode() { - return lockMode; - } - - public void setLockMode(String lockMode) { - this.lockMode = lockMode; - } - - public String getLockExpires() { - return lockExpires; - } - - public void setLockExpires(String lockExpires) { - this.lockExpires = lockExpires; - } - - public String getLegalHold() { - return legalHold; - } - - public void setLegalHold(String legalHold) { - this.legalHold = legalHold; - } -} \ No newline at end of file diff --git a/core/src/com/pspace/ifs/ksan/gw/identity/S3Parameter.java b/core/src/com/pspace/ifs/ksan/gw/identity/S3Parameter.java index dc426f24..b047bc83 100644 --- a/core/src/com/pspace/ifs/ksan/gw/identity/S3Parameter.java +++ b/core/src/com/pspace/ifs/ksan/gw/identity/S3Parameter.java @@ -59,6 +59,7 @@ public class S3Parameter { private String signVersion; private String uri; private boolean isAdmin; + private String taggingInfo; public S3Parameter() { request = null; @@ -80,10 +81,13 @@ public S3Parameter() { fileSize = 0L; uploadId = ""; isWebsite = false; + isPublicAccess = false; statusCode = 0; requestSize = 0L; responseSize = 0L; uri = ""; + isAdmin = false; + taggingInfo = ""; } public HttpServletRequest getRequest() { @@ -421,4 +425,12 @@ public boolean isAdmin() { public void setAdmin(boolean isAdmin) { this.isAdmin = isAdmin; } + + public String getTaggingInfo() { + return taggingInfo; + } + + public void setTaggingInfo(String taggingInfo) { + this.taggingInfo = taggingInfo; + } } diff --git a/core/src/com/pspace/ifs/ksan/gw/identity/S3User.java b/core/src/com/pspace/ifs/ksan/gw/identity/S3User.java index f75965f7..0684018b 100644 --- a/core/src/com/pspace/ifs/ksan/gw/identity/S3User.java +++ b/core/src/com/pspace/ifs/ksan/gw/identity/S3User.java @@ -41,6 +41,7 @@ public class S3User { private String userEmail; private String accessKey; private String accessSecret; + private String azureKey; private List> userDiskPools; public S3User() { @@ -49,6 +50,7 @@ public S3User() { userEmail = ""; accessKey = ""; accessSecret = ""; + azureKey = ""; userDiskPools = new ArrayList>(); } @@ -58,14 +60,22 @@ public S3User(String id, String name, String email, String access, String secret this.userEmail = email; this.accessKey = access; this.accessSecret = secret; + this.azureKey = new String(java.util.Base64.getEncoder().encode(access.getBytes())); userDiskPools = new ArrayList>(); for (int i = 0; i < diskpools.size(); i++) { JSONObject item = (JSONObject)diskpools.get(i); HashMap map = new HashMap(); map.put(USER_DISK_POOLS_DISKPOOL_ID, (String)item.get(S3User.USER_DISK_POOLS_DISKPOOL_ID)); map.put(USER_DISK_POOLS_STORAGE_CLASS, (String)item.get(S3User.USER_DISK_POOLS_STORAGE_CLASS)); + // logger.debug("DiskPoolId:{}, StorageClass : {}", (String)item.get(S3User.USER_DISK_POOLS_DISKPOOL_ID), (String)item.get(S3User.USER_DISK_POOLS_STORAGE_CLASS)); userDiskPools.add(map); } + // logger.debug("getUserDefaultDiskpoolId:{}", getUserDefaultDiskpoolId()); + // logger.debug("userDiskPools:{}", userDiskPools.toString()); + // for (HashMap map : userDiskPools) { + // logger.debug("map:{}", map.toString()); + // logger.debug("DiskPoolId:{}, StorageClass : {}", map.get(USER_DISK_POOLS_DISKPOOL_ID), map.get(USER_DISK_POOLS_STORAGE_CLASS)); + // } } public String getUserName() { @@ -98,6 +108,7 @@ public String getAccessKey() { public void setAccessKey(String accessKey) { this.accessKey = accessKey; + this.azureKey = new String(java.util.Base64.getEncoder().encode(accessKey.getBytes())); } public String getAccessSecret() { @@ -108,13 +119,17 @@ public void setAccessSecret(String accessSecret) { this.accessSecret = accessSecret; } + public String getAzureKey() { + return Strings.nullToEmpty(azureKey); + } + public List> getUserDiskpools() { return userDiskPools; } public String getUserDefaultDiskpoolId() { for (HashMap map : userDiskPools) { - if (map.get(USER_DISK_POOLS_STORAGE_CLASS).equals(STANDARD)) { + if (map.get(USER_DISK_POOLS_STORAGE_CLASS).equalsIgnoreCase(STANDARD)) { return map.get(USER_DISK_POOLS_DISKPOOL_ID); } } @@ -123,7 +138,7 @@ public String getUserDefaultDiskpoolId() { public String getUserDiskpoolId(String storageClass) { for (HashMap map : userDiskPools) { - if (map.get(USER_DISK_POOLS_STORAGE_CLASS).equals(storageClass)) { + if (map.get(USER_DISK_POOLS_STORAGE_CLASS).equalsIgnoreCase(storageClass)) { return map.get(USER_DISK_POOLS_DISKPOOL_ID); } } diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanAbortMultipartUpload.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanAbortMultipartUpload.java index e38a1ca0..a55b02bb 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanAbortMultipartUpload.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanAbortMultipartUpload.java @@ -46,23 +46,8 @@ public void process() throws GWException { initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - DataAbortMultipartUpload dataAbortMultipartUpload = new DataAbortMultipartUpload(s3Parameter); dataAbortMultipartUpload.extract(); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanCompleteMultipartUpload.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanCompleteMultipartUpload.java index 7c954c9c..b8e068f0 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanCompleteMultipartUpload.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanCompleteMultipartUpload.java @@ -25,6 +25,7 @@ import javax.xml.stream.XMLStreamException; import javax.xml.stream.XMLStreamWriter; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.JsonMappingException; @@ -47,6 +48,7 @@ import com.pspace.ifs.ksan.libs.multipart.Multipart; import com.pspace.ifs.ksan.libs.multipart.Part; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; import com.pspace.ifs.ksan.gw.utils.S3UserManager; @@ -64,34 +66,21 @@ public KsanCompleteMultipartUpload(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_COMPLETE_MULTIPART_UPLOAD_START); + logger.info(GWConstants.LOG_ADMIN_COMPLETE_MULTIPART_UPLOAD_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - DataCompleteMultipartUpload dataCompleteMultipartUpload = new DataCompleteMultipartUpload(s3Parameter); dataCompleteMultipartUpload.extract(); String uploadId = dataCompleteMultipartUpload.getUploadId(); s3Parameter.setUploadId(uploadId); + String repVersionId = dataCompleteMultipartUpload.getVersionId(); String multipartXml = dataCompleteMultipartUpload.getMultipartXml(); XmlMapper xmlMapper = new XmlMapper(); @@ -195,27 +184,20 @@ public void process() throws GWException { // check bucket versioning, and set versionId String versioningStatus = getBucketVersioning(bucket); - String versionId = null; Metadata objMeta = null; try { - if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { - versionId = String.valueOf(System.nanoTime()); - objMeta = createLocal(multipart.getDiskPoolId(), bucket, object, versionId); - } else { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - objMeta = createLocal(multipart.getDiskPoolId(), bucket, object, versionId); - } + objMeta = createLocal(multipart.getDiskPoolId(), bucket, object, repVersionId); } catch (GWException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { - logger.info(GWConstants.LOG_COMPLETE_MULTIPART_VERSION_ID, versionId); - s3Parameter.getResponse().addHeader(GWConstants.X_AMZ_VERSION_ID, versionId); + logger.info(GWConstants.LOG_COMPLETE_MULTIPART_VERSION_ID, repVersionId); + s3Parameter.getResponse().addHeader(GWConstants.X_AMZ_VERSION_ID, repVersionId); } - s3Parameter.getResponse().setCharacterEncoding(GWConstants.CHARSET_UTF_8); + s3Parameter.getResponse().setCharacterEncoding(Constants.CHARSET_UTF_8); XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance(); try (Writer writer = s3Parameter.getResponse().getWriter()) { s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); @@ -223,9 +205,9 @@ public void process() throws GWException { final AtomicReference s3Object = new AtomicReference<>(); final AtomicReference S3Excp = new AtomicReference<>(); - S3ObjectOperation objectOperation = new S3ObjectOperation(objMeta, s3Metadata, s3Parameter, versionId, s3ObjectEncryption); + S3ObjectOperation objectOperation = new S3ObjectOperation(objMeta, s3Metadata, s3Parameter, repVersionId, s3ObjectEncryption); - ObjectMapper jsonMapper = new ObjectMapper(); + // ObjectMapper jsonMapper = new ObjectMapper(); SortedMap constListPart = listPart; @@ -296,7 +278,8 @@ public void run() { String jsonmeta = ""; try { - jsonmeta = jsonMapper.writeValueAsString(s3Metadata); + objectMapper.setSerializationInclusion(Include.NON_NULL); + jsonmeta = objectMapper.writeValueAsString(s3Metadata); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); @@ -308,7 +291,7 @@ public void run() { remove(bucket, object); } objMeta.set(s3Object.get().getEtag(), "", jsonmeta, acl, s3Object.get().getFileSize()); - objMeta.setVersionId(versionId, GWConstants.OBJECT_TYPE_FILE, true); + objMeta.setVersionId(repVersionId, GWConstants.OBJECT_TYPE_FILE, true); result = insertObject(bucket, object, objMeta); } catch (GWException e) { PrintStack.logging(logger, e); @@ -317,7 +300,7 @@ public void run() { if (result != 0) { logger.error(GWConstants.LOG_COMPLETE_MULTIPART_UPLOAD_FAILED, bucket, object); } - logger.debug(GWConstants.LOG_COMPLETE_MULTIPART_UPLOAD_INFO, bucket, object, s3Object.get().getFileSize(), s3Object.get().getEtag(), acl, versionId); + logger.debug(GWConstants.LOG_COMPLETE_MULTIPART_UPLOAD_INFO, bucket, object, s3Object.get().getFileSize(), s3Object.get().getEtag(), acl, repVersionId); objMultipart.abortMultipartUpload(uploadId); } catch (IOException e) { PrintStack.logging(logger, e); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanCreateMultipartUpload.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanCreateMultipartUpload.java index 29e76789..a8c09107 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanCreateMultipartUpload.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanCreateMultipartUpload.java @@ -22,6 +22,7 @@ import javax.xml.stream.XMLStreamWriter; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.google.common.base.Strings; import com.pspace.ifs.ksan.gw.data.DataCreateMultipartUpload; @@ -52,31 +53,15 @@ public KsanCreateMultipartUpload(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_CREATE_MULTIPART_UPLOAD_START); + logger.info(GWConstants.LOG_ADMIN_CREATE_MULTIPART_UPLOAD_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); - DataCreateMultipartUpload dataCreateMultipartUpload = new DataCreateMultipartUpload(s3Parameter); dataCreateMultipartUpload.extract(); @@ -87,20 +72,36 @@ public void process() throws GWException { accessControlPolicy.owner.id = s3Parameter.getUser().getUserId(); accessControlPolicy.owner.displayName = s3Parameter.getUser().getUserName(); - String xml = GWUtils.makeAclXml(accessControlPolicy, + // String xml = getBucketInfo().getAcl(); + // String xml = GWUtils.makeAclXml(accessControlPolicy, + // null, + // dataCreateMultipartUpload.hasAclKeyword(), + // null, + // dataCreateMultipartUpload.getAcl(), + // getBucketInfo(), + // getBucketInfo().getUserId(), // s3Parameter.getUser().getUserId(), + // getBucketInfo().getUserName(), // s3Parameter.getUser().getUserName(), + // dataCreateMultipartUpload.getGrantRead(), + // dataCreateMultipartUpload.getGrantWrite(), + // dataCreateMultipartUpload.getGrantFullControl(), + // dataCreateMultipartUpload.getGrantReadAcp(), + // dataCreateMultipartUpload.getGrantWriteAcp(), + // s3Parameter, + // false); + String xml = GWUtils.makeAdmAclXml(accessControlPolicy, null, dataCreateMultipartUpload.hasAclKeyword(), null, dataCreateMultipartUpload.getAcl(), getBucketInfo(), - s3Parameter.getUser().getUserId(), - s3Parameter.getUser().getUserName(), + getBucketInfo().getUserId(), // s3Parameter.getUser().getUserId(), + getBucketInfo().getUserName(), // s3Parameter.getUser().getUserName(), dataCreateMultipartUpload.getGrantRead(), dataCreateMultipartUpload.getGrantWrite(), dataCreateMultipartUpload.getGrantFullControl(), dataCreateMultipartUpload.getGrantReadAcp(), dataCreateMultipartUpload.getGrantWriteAcp(), - s3Parameter); + s3Parameter); String customerAlgorithm = dataCreateMultipartUpload.getServerSideEncryptionCustomerAlgorithm(); String customerKey = dataCreateMultipartUpload.getServerSideEncryptionCustomerKey(); @@ -122,8 +123,8 @@ public void process() throws GWException { logger.debug("storage class : {}, diskpoolId : {}", storageClass, diskpoolId); S3Metadata s3Metadata = new S3Metadata(); - s3Metadata.setOwnerId(s3Parameter.getUser().getUserId()); - s3Metadata.setOwnerName(s3Parameter.getUser().getUserName()); + s3Metadata.setOwnerId(getBucketInfo().getUserId()); + s3Metadata.setOwnerName(getBucketInfo().getUserName()); s3Metadata.setServersideEncryption(serverSideEncryption); s3Metadata.setCustomerAlgorithm(customerAlgorithm); s3Metadata.setCustomerKey(customerKey); @@ -178,6 +179,7 @@ public void process() throws GWException { ObjectMapper jsonMapper = new ObjectMapper(); String metaJson = ""; try { + // jsonMapper.setSerializationInclusion(Include.NON_NULL); metaJson = jsonMapper.writeValueAsString(s3Metadata); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); @@ -198,7 +200,6 @@ public void process() throws GWException { try { ObjMultipart objMultipart = getInstanceObjMultipart(bucket); uploadId = objMultipart.createMultipartUpload(bucket, object, xml, metaJson, objMeta.getPrimaryDisk().getId()); - // uploadId = objMultipart.createMultipartUpload(bucket, object, xml, metaJson, objMeta.getPrimaryDisk().getId()); } catch (Exception e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.INTERNAL_SERVER_ERROR, s3Parameter); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanDeleteObject.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanDeleteObject.java index fd44891e..be34686d 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanDeleteObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanDeleteObject.java @@ -16,6 +16,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.google.common.base.Strings; import com.pspace.ifs.ksan.gw.data.DataDeleteObject; @@ -43,7 +44,7 @@ public KsanDeleteObject(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_DELETE_OBJECT_START); + logger.info(GWConstants.LOG_ADMIN_DELETE_OBJECT_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); @@ -51,23 +52,7 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); logger.debug(GWConstants.LOG_DELETE_OBJECT, bucket, object); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); - - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - - checkGrantBucketOwner(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); DataDeleteObject dataDeleteObject = new DataDeleteObject(s3Parameter); dataDeleteObject.extract(); @@ -175,6 +160,7 @@ private void putDeleteMarker(String bucket, String object, S3Metadata s3Metadata ObjectMapper jsonMapper = new ObjectMapper(); String jsonmeta = ""; + // jsonMapper.setSerializationInclusion(Include.NON_NULL); jsonmeta = jsonMapper.writeValueAsString(s3Metadata); int result; objMeta.set("", "", jsonmeta, "", 0L); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanDeleteObjectTagging.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanDeleteObjectTagging.java index 39b45066..f4bd1b4f 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanDeleteObjectTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanDeleteObjectTagging.java @@ -35,28 +35,14 @@ public KsanDeleteObjectTagging(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_DELETE_OBJECT_TAGGING_START); + logger.info(GWConstants.LOG_ADMIN_DELETE_OBJECT_TAGGING_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - DataDeleteObjectTagging dataDeleteObjectTagging = new DataDeleteObjectTagging(s3Parameter); dataDeleteObjectTagging.extract(); @@ -69,9 +55,6 @@ public void process() throws GWException { } else { objMeta = open(bucket, object, versionId); } - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); - - checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); objMeta.setTag(""); updateObjectTagging(objMeta); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetBucketAcl.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetBucketAcl.java new file mode 100644 index 00000000..36906ac9 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetBucketAcl.java @@ -0,0 +1,66 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.ksanapi; + +import com.pspace.ifs.ksan.gw.api.S3Request; +import java.io.IOException; + +import jakarta.servlet.http.HttpServletResponse; + +import com.google.common.base.Strings; +import com.pspace.ifs.ksan.gw.data.DataGetObjectAcl; +import com.pspace.ifs.ksan.gw.exception.GWErrorCode; +import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.gw.identity.S3Bucket; +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; + +import org.slf4j.LoggerFactory; + +public class KsanGetBucketAcl extends S3Request { + public KsanGetBucketAcl(S3Parameter s3Parameter) { + super(s3Parameter); + logger = LoggerFactory.getLogger(KsanGetBucketAcl.class); + } + + @Override + public void process() throws GWException { + logger.info(GWConstants.LOG_ADMIN_GET_BUCKET_ACL_START); + + String bucket = s3Parameter.getBucketName(); + initBucketInfo(bucket); + + GWUtils.checkCors(s3Parameter); + + DataGetObjectAcl dataGetObjectAcl = new DataGetObjectAcl(s3Parameter); + dataGetObjectAcl.extract(); + + String aclInfo = getBucketInfo().getAcl(); + logger.debug(GWConstants.LOG_ACL, aclInfo); + if (!aclInfo.contains(GWConstants.XML_VERSION)) { + aclInfo = GWConstants.XML_VERSION_FULL_STANDALONE + aclInfo; + } + + try { + if (!Strings.isNullOrEmpty(aclInfo)) { + s3Parameter.getResponse().setContentType(GWConstants.XML_CONTENT_TYPE); + s3Parameter.getResponse().getOutputStream().write(aclInfo.getBytes()); + } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObject.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObject.java index 370e99d9..763fc3bd 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObject.java @@ -50,29 +50,15 @@ public KsanGetObject(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_GET_OBJECT_START); + logger.info(GWConstants.LOG_ADMIN_GET_OBJECT_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); String object = s3Parameter.getObjectName(); logger.debug(GWConstants.LOG_BUCKET_OBJECT, bucket, object); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - DataGetObject dataGetObject = new DataGetObject(s3Parameter); dataGetObject.extract(); @@ -92,8 +78,6 @@ public void process() throws GWException { } logger.debug(GWConstants.LOG_OBJECT_META, objMeta.toString()); - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); - checkGrantObject(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); S3Metadata s3Metadata = null; diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObjectAcl.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObjectAcl.java index 58e5dc31..5f79983d 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObjectAcl.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObjectAcl.java @@ -38,29 +38,15 @@ public KsanGetObjectAcl(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_GET_OBJECT_ACL_START); + logger.info(GWConstants.LOG_ADMIN_GET_OBJECT_ACL_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); String object = s3Parameter.getObjectName(); logger.debug(GWConstants.LOG_BUCKET_OBJECT, bucket, object); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - DataGetObjectAcl dataGetObjectAcl = new DataGetObjectAcl(s3Parameter); dataGetObjectAcl.extract(); String versionId = dataGetObjectAcl.getVersionId(); @@ -72,9 +58,6 @@ public void process() throws GWException { objMeta = open(bucket, object, versionId); } logger.debug(GWConstants.LOG_OBJECT_META, objMeta.toString()); - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); - - checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ_ACP); String aclInfo = objMeta.getAcl(); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObjectTagging.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObjectTagging.java index 142f9018..3afb859a 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObjectTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanGetObjectTagging.java @@ -40,30 +40,17 @@ public KsanGetObjectTagging(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_GET_OBJECT_TAGGING_START); + logger.info(GWConstants.LOG_ADMIN_GET_OBJECT_TAGGING_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); - GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); + String object = s3Parameter.getObjectName(); + GWUtils.checkCors(s3Parameter); - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } DataGetObjectTagging dataGetObjectTagging = new DataGetObjectTagging(s3Parameter); dataGetObjectTagging.extract(); - - String object = s3Parameter.getObjectName(); - + String versionId = dataGetObjectTagging.getVersionId(); Metadata objMeta = null; @@ -72,10 +59,6 @@ public void process() throws GWException { } else { objMeta = open(bucket, object, versionId); } - - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); - - checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); String taggingInfo = objMeta.getTag(); logger.info(GWConstants.LOG_TAGGING, taggingInfo); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanHeadObject.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanHeadObject.java index b5ee4d18..68ab165f 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanHeadObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanHeadObject.java @@ -10,12 +10,21 @@ */ package com.pspace.ifs.ksan.gw.ksanapi; import com.pspace.ifs.ksan.gw.api.S3Request; +import com.pspace.ifs.ksan.gw.api.S3AddResponse; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Date; + +import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletResponse; import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.ObjectMapper; import com.google.common.base.Strings; +import com.google.common.net.HttpHeaders; + import com.pspace.ifs.ksan.gw.data.DataHeadObject; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; @@ -31,7 +40,7 @@ import org.slf4j.LoggerFactory; -public class KsanHeadObject extends S3Request { +public class KsanHeadObject extends S3Request implements S3AddResponse { public KsanHeadObject(S3Parameter s3Parameter) { super(s3Parameter); logger = LoggerFactory.getLogger(KsanHeadObject.class); @@ -39,36 +48,21 @@ public KsanHeadObject(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_HEAD_OBJECT_START); + logger.info(GWConstants.LOG_ADMIN_HEAD_OBJECT_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); String object = s3Parameter.getObjectName(); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); - GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - + GWUtils.checkCors(s3Parameter); + DataHeadObject dataHeadObject = new DataHeadObject(s3Parameter); dataHeadObject.extract(); String versionId = dataHeadObject.getVersionId(); String expectedBucketOwner = dataHeadObject.getExpectedBucketOwner(); - if (!Strings.isNullOrEmpty(expectedBucketOwner)) { - if (!isBucketOwner(expectedBucketOwner)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } + if (!Strings.isNullOrEmpty(expectedBucketOwner) && !isBucketOwner(expectedBucketOwner)) { + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } Metadata objMeta = null; @@ -79,9 +73,6 @@ public void process() throws GWException { objMeta = open(bucket, object, versionId); } - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); - checkGrantObject(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_READ); - // meta info ObjectMapper objectMapper = new ObjectMapper(); try { @@ -101,13 +92,130 @@ public void process() throws GWException { } } + // s3Parameter.getResponse().addHeader(GWConstants.X_AMZ_VERSION_ID, s3Metadata.getVersionId()); + // GWUtils.addMetadataToResponse(s3Parameter.getRequest(), s3Parameter.getResponse(), s3Metadata, null, null); + s3Parameter.getResponse().addHeader(GWConstants.X_AMZ_VERSION_ID, s3Metadata.getVersionId()); - GWUtils.addMetadataToResponse(s3Parameter.getRequest(), s3Parameter.getResponse(), s3Metadata, null, null); + addMetadataToResponse(s3Parameter.getResponse(), s3Metadata, null, null); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } - s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + s3Parameter.getResponse().setStatus(HttpServletResponse.SC_OK); + } + + @Override + public void addResponseHeaderWithOverride(HttpServletRequest request, HttpServletResponse response, + String headerName, String overrideHeaderName, String value) { + + String override = request.getParameter(overrideHeaderName); + + // NPE in if value is null + override = (!Strings.isNullOrEmpty(override)) ? override : value; + + if (!Strings.isNullOrEmpty(override)) { + response.addHeader(headerName, override); + } + } + + @Override + public void addMetadataToResponse(HttpServletResponse response, S3Metadata metadata, List contentsHeaders, + Long streamSize) { + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CACHE_CONTROL, GWConstants.RESPONSE_CACHE_CONTROL, + metadata.getCacheControl()); + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CONTENT_ENCODING, GWConstants.RESPONSE_CONTENT_ENCODING, + metadata.getContentEncoding()); + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CONTENT_LANGUAGE, GWConstants.RESPONSE_CONTENT_LANGUAGE, + metadata.getContentLanguage()); + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CONTENT_DISPOSITION, GWConstants.RESPONSE_CONTENT_DISPOSITION, + metadata.getContentDisposition()); + addResponseHeaderWithOverride(s3Parameter.getRequest(), response, + HttpHeaders.CONTENT_TYPE, GWConstants.RESPONSE_CONTENT_TYPE, + metadata.getContentType()); + + Collection contentRanges = contentsHeaders; + if (contentsHeaders != null && !contentRanges.isEmpty()) { + for (String contents : contentsHeaders) { + response.addHeader(HttpHeaders.CONTENT_RANGE, contents); + } + + response.addHeader(HttpHeaders.ACCEPT_RANGES, GWConstants.BYTES); + response.addHeader(HttpHeaders.CONTENT_LENGTH, streamSize.toString()); + } else { + response.addHeader(HttpHeaders.CONTENT_LENGTH, metadata.getContentLength().toString()); + logger.debug(GWConstants.LOG_GET_OBJECT_CONTENT_LENGTH, metadata.getContentLength()); + } + + String overrideContentType = s3Parameter.getRequest().getParameter(GWConstants.RESPONSE_CONTENT_TYPE); + response.setContentType(overrideContentType != null ? overrideContentType : metadata.getContentType()); + + if (!Strings.isNullOrEmpty(metadata.getCustomerAlgorithm())) { + response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, metadata.getCustomerAlgorithm()); + } + + if (!Strings.isNullOrEmpty(metadata.getCustomerKey())) { + response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, metadata.getCustomerKey()); + } + + if (!Strings.isNullOrEmpty(metadata.getCustomerKeyMD5())) { + response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, metadata.getCustomerKeyMD5()); + } + + if (!Strings.isNullOrEmpty(metadata.getServersideEncryption())) { + response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION, metadata.getServersideEncryption()); + } + + if (!Strings.isNullOrEmpty(metadata.getLockMode())) { + response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_MODE, metadata.getLockMode()); + } + + if (!Strings.isNullOrEmpty(metadata.getLockExpires())) { + response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, metadata.getLockExpires()); + } + + if (!Strings.isNullOrEmpty(metadata.getLegalHold())) { + response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_LEGAL_HOLD, metadata.getLegalHold()); + } + + if (metadata.getUserMetadataMap() != null) { + for (Map.Entry entry : metadata.getUserMetadataMap().entrySet()) { + response.addHeader(entry.getKey(), entry.getValue()); + logger.debug(GWConstants.LOG_GET_OBJECT_USER_META_DATA, entry.getKey(), entry.getValue()); + } + } + + response.addHeader(HttpHeaders.ETAG, GWUtils.maybeQuoteETag(metadata.getETag())); + + String overrideExpires = s3Parameter.getRequest().getParameter(GWConstants.RESPONSE_EXPIRES); + if (overrideExpires != null) { + response.addHeader(HttpHeaders.EXPIRES, overrideExpires); + } else { + Date expires = metadata.getExpires(); + if (expires != null) { + response.addDateHeader(HttpHeaders.EXPIRES, expires.getTime()); + } + } + + logger.debug(GWConstants.LOG_GET_OBJECT_MODIFIED, metadata.getLastModified().getTime()); + response.addDateHeader(HttpHeaders.LAST_MODIFIED, metadata.getLastModified().getTime()); + + if (!Strings.isNullOrEmpty(metadata.getTaggingCount())) { + response.addHeader(GWConstants.X_AMZ_TAGGING_COUNT, metadata.getTaggingCount()); + } + + if (!Strings.isNullOrEmpty(metadata.getVersionId())) { + if (metadata.getVersionId().equalsIgnoreCase(GWConstants.VERSIONING_DISABLE_TAIL)) { + response.addHeader(GWConstants.X_AMZ_VERSION_ID, GWConstants.VERSIONING_DISABLE_TAIL); + } else { + response.addHeader(GWConstants.X_AMZ_VERSION_ID, metadata.getVersionId()); + } + } else { + response.addHeader(GWConstants.X_AMZ_VERSION_ID, GWConstants.VERSIONING_DISABLE_TAIL); + } } } diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObject.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObject.java index 5cacdd32..016a5796 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObject.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObject.java @@ -18,6 +18,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.dataformat.xml.XmlMapper; import com.google.common.base.Strings; @@ -60,7 +61,7 @@ public KsanPutObject(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_PUT_OBJECT_START); + logger.info(GWConstants.LOG_ADMIN_PUT_OBJECT_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); @@ -68,24 +69,14 @@ public void process() throws GWException { String object = s3Parameter.getObjectName(); logger.debug(GWConstants.LOG_BUCKET_OBJECT, bucket, object); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); + S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); if (user == null) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); } s3Parameter.setUser(user); - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); - DataPutObject dataPutObject = new DataPutObject(s3Parameter); dataPutObject.extract(); @@ -104,17 +95,25 @@ public void process() throws GWException { String customerKeyMD5 = dataPutObject.getServerSideEncryptionCustomerKeyMD5(); String serversideEncryption = dataPutObject.getServerSideEncryption(); String storageClass = dataPutObject.getStorageClass(); + String replication = dataPutObject.getReplication(); + String repVersionId = dataPutObject.getVersionId(); + + String versioningStatus = getBucketVersioning(bucket); + if (!Strings.isNullOrEmpty(replication)) { + if (!GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { + throw new GWException(GWErrorCode.BAD_REQUEST, s3Parameter); + } + } if (Strings.isNullOrEmpty(storageClass)) { storageClass = GWConstants.AWS_TIER_STANTARD; } - String diskpoolId = s3Parameter.getUser().getUserDiskpoolId(storageClass); logger.debug("storage class : {}, diskpoolId : {}", storageClass, diskpoolId); - s3Metadata.setOwnerId(s3Parameter.getUser().getUserId()); - s3Metadata.setOwnerName(s3Parameter.getUser().getUserName()); + s3Metadata.setOwnerId(getBucketInfo().getUserId()); + s3Metadata.setOwnerName(getBucketInfo().getUserName()); s3Metadata.setUserMetadataMap(dataPutObject.getUserMetadata()); if (!Strings.isNullOrEmpty(serversideEncryption)) { @@ -191,14 +190,14 @@ public void process() throws GWException { accessControlPolicy.owner.id = s3Parameter.getUser().getUserId(); accessControlPolicy.owner.displayName = s3Parameter.getUser().getUserName(); - String aclXml = GWUtils.makeAclXml(accessControlPolicy, + String aclXml = GWUtils.makeAdmAclXml(accessControlPolicy, null, dataPutObject.hasAclKeyword(), null, dataPutObject.getAcl(), getBucketInfo(), - s3Parameter.getUser().getUserId(), - s3Parameter.getUser().getUserName(), + getBucketInfo().getUserId(), // s3Parameter.getUser().getUserId(), + getBucketInfo().getUserName(), // s3Parameter.getUser().getUserName(), dataPutObject.getGrantRead(), dataPutObject.getGrantWrite(), dataPutObject.getGrantFullControl(), @@ -333,28 +332,18 @@ public void process() throws GWException { s3Metadata.setLegalHold(dataPutObject.getObjectLockLegalHold()); } - String versioningStatus = getBucketVersioning(bucket); - String versionId = null; Metadata objMeta = null; try { // check exist object - objMeta = open(bucket, object); - if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { - versionId = String.valueOf(System.nanoTime()); - } else { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - } + objMeta = open(bucket, object, repVersionId); } catch (GWException e) { logger.info(e.getMessage()); - if (GWConstants.VERSIONING_ENABLED.equalsIgnoreCase(versioningStatus)) { - versionId = String.valueOf(System.nanoTime()); - objMeta = createLocal(diskpoolId, bucket, object, versionId); - } else { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - objMeta = createLocal(diskpoolId, bucket, object, versionId); - } + s3Parameter.setErrorCode(GWConstants.EMPTY_STRING); + s3Parameter.setStatusCode(0); + objMeta = createLocal(diskpoolId, bucket, object, repVersionId); } - S3ObjectOperation objectOperation = new S3ObjectOperation(objMeta, s3Metadata, s3Parameter, versionId, encryption); + s3Parameter.setVersionId(repVersionId); + S3ObjectOperation objectOperation = new S3ObjectOperation(objMeta, s3Metadata, s3Parameter, repVersionId, encryption); S3Object s3Object = objectOperation.putObject(); s3Metadata.setETag(s3Object.getEtag()); @@ -373,6 +362,7 @@ public void process() throws GWException { ObjectMapper jsonMapper = new ObjectMapper(); String jsonmeta = ""; try { + // jsonMapper.setSerializationInclusion(Include.NON_NULL); jsonmeta = jsonMapper.writeValueAsString(s3Metadata); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); @@ -382,9 +372,9 @@ public void process() throws GWException { logger.debug(GWConstants.LOG_PUT_OBJECT_PRIMARY_DISK_ID, objMeta.getPrimaryDisk().getId()); try { objMeta.set(s3Object.getEtag(), taggingxml, jsonmeta, aclXml, s3Object.getFileSize()); - objMeta.setVersionId(versionId, GWConstants.OBJECT_TYPE_FILE, true); + objMeta.setVersionId(repVersionId, GWConstants.OBJECT_TYPE_FILE, true); int result = insertObject(bucket, object, objMeta); - logger.debug(GWConstants.LOG_PUT_OBJECT_INFO, bucket, object, s3Object.getFileSize(), s3Object.getEtag(), aclXml, versionId); + logger.debug(GWConstants.LOG_PUT_OBJECT_INFO, bucket, object, s3Object.getFileSize(), s3Object.getEtag(), aclXml, repVersionId); } catch (GWException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObjectAcl.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObjectAcl.java index 838cb399..ebb0ff4c 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObjectAcl.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObjectAcl.java @@ -41,26 +41,13 @@ public KsanPutObjectAcl(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_PUT_OBJECT_ACL_START); + logger.info(GWConstants.LOG_ADMIN_PUT_OBJECT_ACL_START); String bucket = s3Parameter.getBucketName(); String object = s3Parameter.getObjectName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); - GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } + GWUtils.checkCors(s3Parameter); DataPutObjectAcl dataPutObjectAcl = new DataPutObjectAcl(s3Parameter); dataPutObjectAcl.extract(); @@ -74,10 +61,6 @@ public void process() throws GWException { objMeta = open(bucket, object, versionId); } - objMeta.setAcl(GWUtils.makeOriginalXml(objMeta.getAcl(), s3Parameter)); - - checkGrantObjectOwner(s3Parameter.isPublicAccess(), objMeta, s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE_ACP); - accessControlPolicy = new AccessControlPolicy(); accessControlPolicy.aclList = new AccessControlList(); accessControlPolicy.aclList.grants = new ArrayList(); @@ -97,7 +80,8 @@ public void process() throws GWException { dataPutObjectAcl.getGrantFullControl(), dataPutObjectAcl.getGrantReadAcp(), dataPutObjectAcl.getGrantWriteAcp(), - s3Parameter); + s3Parameter, + true); logger.debug(GWConstants.LOG_ACL, xml); objMeta.setAcl(xml); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObjectTagging.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObjectTagging.java index c0cb3a87..8523a91f 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObjectTagging.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanPutObjectTagging.java @@ -16,6 +16,7 @@ import jakarta.servlet.http.HttpServletResponse; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.annotation.JsonInclude.Include; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.dataformat.xml.XmlMapper; import com.google.common.base.Strings; @@ -44,28 +45,11 @@ public KsanPutObjectTagging(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_PUT_OBJECT_TAGGING_START); + logger.info(GWConstants.LOG_ADMIN_PUT_OBJECT_TAGGING_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); - S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); - GWUtils.checkCors(s3Parameter); - - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); - String object = s3Parameter.getObjectName(); + GWUtils.checkCors(s3Parameter); DataPutObjectTagging dataPutObjectTagging = new DataPutObjectTagging(s3Parameter); dataPutObjectTagging.extract(); @@ -124,10 +108,11 @@ public void process() throws GWException { } s3Metadata.setTaggingCount(taggingCount); - ObjectMapper jsonMapper = new ObjectMapper(); + // ObjectMapper jsonMapper = new ObjectMapper(); String jsonMeta = ""; try { - jsonMeta = jsonMapper.writeValueAsString(s3Metadata); + objectMapper.setSerializationInclusion(Include.NON_NULL); + jsonMeta = objectMapper.writeValueAsString(s3Metadata); } catch (JsonProcessingException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); diff --git a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanUploadPart.java b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanUploadPart.java index 205cbff5..fe338cdd 100644 --- a/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanUploadPart.java +++ b/core/src/com/pspace/ifs/ksan/gw/ksanapi/KsanUploadPart.java @@ -48,29 +48,15 @@ public KsanUploadPart(S3Parameter s3Parameter) { @Override public void process() throws GWException { - logger.info(GWConstants.LOG_UPLOAD_PART_START); + logger.info(GWConstants.LOG_ADMIN_UPLOAD_PART_START); String bucket = s3Parameter.getBucketName(); initBucketInfo(bucket); String object = s3Parameter.getObjectName(); S3Bucket s3Bucket = new S3Bucket(); - s3Bucket.setCors(getBucketInfo().getCors()); - s3Bucket.setAccess(getBucketInfo().getAccess()); - s3Parameter.setBucket(s3Bucket); - GWUtils.checkCors(s3Parameter); - S3User user = S3UserManager.getInstance().getUserByName(getBucketInfo().getUserName()); - if (user == null) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - s3Parameter.setUser(user); - - if (s3Parameter.isPublicAccess() && GWUtils.isIgnorePublicAcls(s3Parameter)) { - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - - checkGrantBucket(s3Parameter.isPublicAccess(), s3Parameter.getUser().getUserId(), GWConstants.GRANT_WRITE); + GWUtils.checkCors(s3Parameter); DataUploadPart dataUploadPart = new DataUploadPart(s3Parameter); dataUploadPart.extract(); diff --git a/core/src/com/pspace/ifs/ksan/gw/mq/ChannelFactory.java b/core/src/com/pspace/ifs/ksan/gw/mq/ChannelFactory.java new file mode 100644 index 00000000..0f29bec1 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/mq/ChannelFactory.java @@ -0,0 +1,86 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.mq; + +import org.apache.commons.pool2.PooledObject; +import org.apache.commons.pool2.PooledObjectFactory; +import org.apache.commons.pool2.impl.DefaultPooledObject; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.libs.PrintStack; +import com.rabbitmq.client.Channel; +import com.rabbitmq.client.Connection; +import com.rabbitmq.client.ConnectionFactory; + +public class ChannelFactory implements PooledObjectFactory { + private String host; + private int port; + private String username; + private String password; + private String exchangeName; + private String exchangeOption; + private String routingKey; + + private final static Logger logger = LoggerFactory.getLogger(ChannelFactory.class); + + public ChannelFactory(String host, int port, String username, String password, String exchangeName, String exchangeOption, String routingKey) { + try { + this.host = host; + this.port = port; + this.username = username; + this.password = password; + this.exchangeName = exchangeName;; + this.exchangeOption = exchangeOption; + this.routingKey = routingKey; + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } + + public PooledObject makeObject() throws Exception { + ConnectionFactory factory = new ConnectionFactory(); + factory.setHost(host); + factory.setUsername(username); + factory.setPassword(password); + factory.setPort(port); + factory.setAutomaticRecoveryEnabled(true); + factory.setNetworkRecoveryInterval(10000); + factory.setConnectionTimeout(10000); + Connection connection = factory.newConnection(); + Channel channel = connection.createChannel(); + channel.basicQos(0); + return new DefaultPooledObject(channel); + } + + public void destroyObject(PooledObject pooledObject) throws Exception { + final Channel channel = pooledObject.getObject(); + if (channel.isOpen()) { + try { + channel.close(); + } catch (Exception e) { + } + } + } + + public boolean validateObject(PooledObject pooledObject) { + final Channel channel = pooledObject.getObject(); + return channel.isOpen(); + } + + public void activateObject(PooledObject pooledObject) throws Exception { + + } + + public void passivateObject(PooledObject pooledObject) throws Exception { + + } +} \ No newline at end of file diff --git a/core/src/com/pspace/ifs/ksan/gw/mq/ChannelPool.java b/core/src/com/pspace/ifs/ksan/gw/mq/ChannelPool.java new file mode 100644 index 00000000..d6ebd34d --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/mq/ChannelPool.java @@ -0,0 +1,76 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.mq; + +import java.time.Duration; + +import org.apache.commons.pool2.impl.GenericObjectPool; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.libs.PrintStack; +import com.rabbitmq.client.Channel; + +public class ChannelPool { + private final static Logger logger = LoggerFactory.getLogger(ChannelFactory.class); + private GenericObjectPool internalPool; + + + public ChannelPool(String host, int port, String username, String password, String exchangeName, String exchangeOption, String routingKey) { + this(new ChannelFactory(host, port, username, password, exchangeName, exchangeOption, routingKey)); + } + + public ChannelPool(ChannelFactory factory) { + if (this.internalPool != null) { + try { + closeInternalPool(); + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } + + this.internalPool = new GenericObjectPool(factory); + this.internalPool.setMaxTotal(10); + this.internalPool.setBlockWhenExhausted(true); + this.internalPool.setMaxWait(Duration.ofSeconds(30)); + } + + private void closeInternalPool() { + try { + internalPool.close(); + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } + + public void returnChannel(Channel channel) { + try { + if (channel.isOpen()) { + internalPool.returnObject(channel); + } else { + internalPool.invalidateObject(channel); + } + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } + + public Channel getChannel() { + try { + return internalPool.borrowObject(); + } catch (Exception e) { + PrintStack.logging(logger, e); + } + + return null; + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/mq/MessageQueueSender.java b/core/src/com/pspace/ifs/ksan/gw/mq/MessageQueueSender.java new file mode 100644 index 00000000..bc8db694 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/mq/MessageQueueSender.java @@ -0,0 +1,44 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.mq; + +import com.rabbitmq.client.AMQP.BasicProperties; +import com.rabbitmq.client.Channel; + +/** + * + * @author legesse + */ +public class MessageQueueSender { + ChannelPool channelPool; + String exchangeName; + + public MessageQueueSender(String host, int port, String username, String password, String exchangeName, String exchangeOption, String routingKey) + throws Exception{ + channelPool = new ChannelPool(host, port, username, password, exchangeName, exchangeOption, routingKey); + this.exchangeName = exchangeName; + } + + public int send(String mesg, String routingKey) throws Exception{ + Channel channel = channelPool.getChannel(); + String replyQueueName = channel.queueDeclarePassive("replyQ").getQueue(); + + BasicProperties props = new BasicProperties + .Builder() + .replyTo(replyQueueName) + .build(); + + + channel.basicPublish(this.exchangeName, routingKey, props, mesg.getBytes()); + channelPool.returnChannel(channel); + return 0; + } +} \ No newline at end of file diff --git a/core/src/com/pspace/ifs/ksan/gw/object/AzuObjectOperation.java b/core/src/com/pspace/ifs/ksan/gw/object/AzuObjectOperation.java new file mode 100644 index 00000000..964cae05 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/object/AzuObjectOperation.java @@ -0,0 +1,957 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.object; + +import static com.google.common.io.BaseEncoding.base64; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileNotFoundException; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.RandomAccessFile; +import java.nio.ByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.charset.Charset; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.StandardCopyOption; +import java.nio.file.attribute.UserDefinedFileAttributeView; +import java.security.InvalidKeyException; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.Date; +import java.util.Iterator; +import java.util.Map; +import java.util.SortedMap; +import java.util.UUID; +import java.util.Collection; + +import com.google.common.base.Strings; +import com.google.common.primitives.Longs; +import com.pspace.ifs.ksan.gw.exception.AzuErrorCode; +import com.pspace.ifs.ksan.gw.exception.AzuException; +import com.pspace.ifs.ksan.libs.Constants; +import com.pspace.ifs.ksan.libs.identity.S3Metadata; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.libs.multipart.Part; +import com.pspace.ifs.ksan.gw.object.objmanager.ObjManagerHelper; +import com.pspace.ifs.ksan.gw.object.osdclient.OSDClientManager; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.GWConfig; +import com.pspace.ifs.ksan.gw.utils.GWConstants; +import com.pspace.ifs.ksan.gw.utils.GWUtils; +import com.pspace.ifs.ksan.objmanager.Metadata; +import com.pspace.ifs.ksan.objmanager.ObjManager; +import com.pspace.ifs.ksan.objmanager.ObjManagerException.ResourceNotFoundException; +import com.pspace.ifs.ksan.libs.DiskManager; +import com.pspace.ifs.ksan.libs.OSDClient; +import com.pspace.ifs.ksan.libs.data.OsdData; +import com.pspace.ifs.ksan.libs.KsanUtils; + +import de.sfuhrm.openssl4j.OpenSSL4JProvider; + +import org.apache.commons.crypto.stream.CtrCryptoInputStream; +import org.apache.commons.crypto.stream.CtrCryptoOutputStream; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class AzuObjectOperation { + private Metadata objMeta; + private S3Metadata s3Meta; + private AzuParameter azuParameter; + private String versionId; + private ObjManager objManager; + private static final Logger logger = LoggerFactory.getLogger(AzuObjectOperation.class); + + public AzuObjectOperation(Metadata objMeta, S3Metadata s3Meta, AzuParameter azuParameter, String versionId) { + this.objMeta = objMeta; + this.s3Meta = s3Meta; + this.azuParameter = azuParameter; + + if (Strings.isNullOrEmpty(versionId)) { + this.versionId = GWConstants.VERSIONING_DISABLE_TAIL; + } else { + this.versionId = versionId; + } + } + + public void getObject(String range) throws Exception { + OSDClient client = null; + String sourceRange = AzuConstants.EMPTY_STRING; + long actualSize = 0L; + long fileSize = objMeta.getSize(); + + try { + if (objMeta.getReplicaCount() > 1) { + logger.debug("bucket : {}, object : {}", objMeta.getBucket(), objMeta.getPath()); + logger.debug("primary disk id : {}, osd ip : {}", objMeta.getPrimaryDisk().getId(), objMeta.getPrimaryDisk().getOsdIp()); + if (objMeta.isReplicaExist()) { + logger.debug("replica disk id : {}, osd ip : {}", objMeta.getReplicaDisk().getId(), objMeta.getReplicaDisk().getOsdIp()); + } + + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + logger.debug("get local - objid : {}, primary get - path : {}", objMeta.getObjId(), objMeta.getPrimaryDisk().getPath()); + actualSize = getObjectLocal(azuParameter.getResponse().getOutputStream(), objMeta.getPrimaryDisk().getPath(), range, objMeta.getObjId()); + azuParameter.addResponseSize(actualSize); + } else if (objMeta.isReplicaExist() && GWUtils.getLocalIP().equals(objMeta.getReplicaDisk().getOsdIp())) { + logger.debug("get local - objid : {}, replica get - path : {}", objMeta.getObjId(), objMeta.getReplicaDisk().getPath()); + actualSize = getObjectLocal(azuParameter.getResponse().getOutputStream(), objMeta.getReplicaDisk().getPath(), range, objMeta.getObjId()); + azuParameter.addResponseSize(actualSize); + } else { + logger.debug("get osd - objid : {}, primary osd : {}", objMeta.getObjId(), objMeta.getPrimaryDisk().getOsdIp()); + try { + // client = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (client == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // client = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + client = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + client.getInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + objMeta.getVersionId(), + fileSize, + sourceRange, + azuParameter.getResponse().getOutputStream(), + AzuConstants.EMPTY_STRING); + actualSize = client.get(); + // OSDClientManager.getInstance().returnOSDClient(client); + azuParameter.addResponseSize(actualSize); + } catch (Exception e) { + PrintStack.logging(logger, e); + if (objMeta.isReplicaExist()) { + try { + // client = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // if (client == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getReplicaDisk().getOsdIp()); + // client = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // } + client = new OSDClient(objMeta.getReplicaDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + client.getInit(objMeta.getReplicaDisk().getPath(), + objMeta.getObjId(), + objMeta.getVersionId(), + fileSize, + sourceRange, + azuParameter.getResponse().getOutputStream(), + AzuConstants.EMPTY_STRING); + actualSize = client.get(); + // OSDClientManager.getInstance().returnOSDClient(client); + client = null; + azuParameter.addResponseSize(actualSize); + } catch (Exception e1) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } else { + // Can't find the object + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + } + } else { + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + actualSize = getObjectLocal(azuParameter.getResponse().getOutputStream(), objMeta.getPrimaryDisk().getPath(), range, objMeta.getObjId()); + azuParameter.addResponseSize(actualSize); + } else { + try { + // client = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (client == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // client = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + client = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + client.getInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + objMeta.getVersionId(), + fileSize, + sourceRange, + azuParameter.getResponse().getOutputStream(), + AzuConstants.EMPTY_STRING); + actualSize = client.get(); + // OSDClientManager.getInstance().returnOSDClient(client); + client = null; + azuParameter.addResponseSize(actualSize); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + } + } catch (Exception e) { + PrintStack.logging(logger, e); + if (client != null) { + // OSDClientManager.getInstance().returnOSDClient(client); + } + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_FILE_SIZE, actualSize); + } + + private long getObjectLocal(OutputStream outputStream, String path, String range, String objId) throws IOException, AzuException { + byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; + File file = null; + + file = new File(KsanUtils.makeObjPath(path, objId, versionId)); + + logger.info("obj path : {}", file.getAbsolutePath()); + long actualSize = 0L; + + try (FileInputStream fis = new FileInputStream(file)) { + long remaingLength = 0L; + int readLength = 0; + int readBytes; + remaingLength = file.length(); + + if (Strings.isNullOrEmpty(range)) { + while (remaingLength > 0) { + readBytes = 0; + if (remaingLength < GWConstants.MAXBUFSIZE) { + readBytes = (int)remaingLength; + } else { + readBytes = GWConstants.MAXBUFSIZE; + } + + if (remaingLength >= GWConstants.MAXBUFSIZE) { + readLength = GWConstants.MAXBUFSIZE; + } else { + readLength = (int)remaingLength; + } + readLength = fis.read(buffer, 0, readBytes); + + actualSize += readLength; + outputStream.write(buffer, 0, readLength); + // s3Parameter.addResponseSize(readLength); + remaingLength -= readLength; + } + } else { + String[] infos = range.split(GWConstants.EQUAL); + String[] ranges = infos[1].split(GWConstants.DASH); + long offset = Longs.tryParse(ranges[0]); + long length = Longs.tryParse(ranges[1]); + logger.info("offset : {}, length : {}", offset, length); + remaingLength = length - offset + 1; + if (offset > 0) { + fis.skip(offset); + } + while (remaingLength > 0) { + readBytes = 0; + if (remaingLength < GWConstants.MAXBUFSIZE) { + readBytes = (int)remaingLength; + } else { + readBytes = GWConstants.MAXBUFSIZE; + } + + readLength = fis.read(buffer, 0, readBytes); + + actualSize += readLength; + outputStream.write(buffer, 0, readLength); + remaingLength -= readLength; + } + } + } + + outputStream.flush(); + outputStream.close(); + + return actualSize; + } + + public S3Object putObject() throws AzuException { + S3Object s3Object = new S3Object(); + + File filePrimary = null; + File tmpFilePrimary = null; + FileOutputStream fosPrimary = null; + File fileReplica = null; + File tmpFileReplica = null; + FileOutputStream fosReplica = null; + File trashPrimary = null; + File trashReplica = null; + OSDClient clientPrimary = null; + OSDClient clientReplica = null; + long length = s3Meta.getContentLength(); + long totalReads = 0L; + long existFileSize = 0L; + long putSize = 0L; + long calSize = 0L; + + try { + MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5, new OpenSSL4JProvider()); + byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; + int readLength = 0; + + existFileSize = objMeta.getSize(); + putSize = length; + boolean isPrimaryCache = false; + boolean isReplicaCache = false; + + logger.debug("performance mode : {}", GWConfig.getInstance().getPerformanceMode()); + logger.debug("objMeta - replicaCount : {}", objMeta.getReplicaCount()); + + + if (objMeta.getReplicaCount() > 1) { + existFileSize *= objMeta.getReplicaCount(); + putSize *= objMeta.getReplicaCount(); + logger.debug("bucket : {}, object : {}", objMeta.getBucket(), objMeta.getPath()); + logger.debug("primary disk id : {}, osd ip : {}", objMeta.getPrimaryDisk().getId(), objMeta.getPrimaryDisk().getOsdIp()); + if (objMeta.isReplicaExist()) { + logger.debug("replica disk id : {}, osd ip : {}", objMeta.getReplicaDisk().getId(), objMeta.getReplicaDisk().getOsdIp()); + } + + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + filePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFilePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + trashPrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + isPrimaryCache = true; + } else { + filePrimary = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFilePrimary = new File(KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + trashPrimary = new File(KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + } + com.google.common.io.Files.createParentDirs(filePrimary); + com.google.common.io.Files.createParentDirs(tmpFilePrimary); + fosPrimary = new FileOutputStream(tmpFilePrimary, false); + } else { + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (clientPrimary == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + clientPrimary = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientPrimary.getSocket().getRemoteSocketAddress().toString(), clientPrimary.getSocket().getLocalPort()); + clientPrimary.putInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + versionId, + length, + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + objMeta.getReplicaDisk().getId(), + GWConstants.EMPTY_STRING, + GWConfig.getInstance().getPerformanceMode()); + } + + if (objMeta.isReplicaExist()) { + if (GWUtils.getLocalIP().equals(objMeta.getReplicaDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + fileReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFileReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); + trashReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); + isReplicaCache = true; + } else { + fileReplica = new File(KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFileReplica = new File(KsanUtils.makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + trashReplica = new File(KsanUtils.makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + } + com.google.common.io.Files.createParentDirs(fileReplica); + com.google.common.io.Files.createParentDirs(tmpFileReplica); + fosReplica = new FileOutputStream(tmpFileReplica, false); + } else { + // clientReplica = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // if (clientReplica == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getReplicaDisk().getOsdIp()); + // clientReplica = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // } + clientReplica = new OSDClient(objMeta.getReplicaDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientReplica.getSocket().getRemoteSocketAddress().toString(), clientReplica.getSocket().getLocalPort()); + clientReplica.putInit(objMeta.getReplicaDisk().getPath(), + objMeta.getObjId(), + versionId, + length, + Constants.FILE_ATTRIBUTE_REPLICATION_REPLICA, + Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, + GWConstants.EMPTY_STRING, + GWConfig.getInstance().getPerformanceMode()); + } + } + + while ((readLength = azuParameter.getInputStream().read(buffer, 0, GWConstants.BUFSIZE)) != -1) { + totalReads += readLength; + if (filePrimary == null) { + clientPrimary.put(buffer, 0, readLength); + } else { + fosPrimary.write(buffer, 0, readLength); + } + + if (objMeta.isReplicaExist()) { + if (fileReplica == null) { + clientReplica.put(buffer, 0, readLength); + } else { + fosReplica.write(buffer, 0, readLength); + } + } + + md5er.update(buffer, 0, readLength); + + if (totalReads >= length) { + break; + } + } + logger.info("total read : {}", totalReads); + + if (filePrimary == null) { + clientPrimary.putFlush(); + // OSDClientManager.getInstance().returnOSDClient(clientPrimary); + clientPrimary = null; + } else { + fosPrimary.flush(); + fosPrimary.close(); + + if (filePrimary.exists()) { + File tempFile = new File(filePrimary.getAbsolutePath()); + logger.debug("filePrimary is already exists : {}", filePrimary.getAbsolutePath()); + retryRenameTo(tempFile, trashPrimary); + } + + // if (objMeta.getReplicaDisk() != null && !Strings.isNullOrEmpty(objMeta.getReplicaDisk().getId())) { + // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId()); + // } else { + // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + // } + + retryRenameTo(tmpFilePrimary, filePrimary); + if (isPrimaryCache) { + String path = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + com.google.common.io.Files.createParentDirs(new File(path)); + logger.debug("path : {}, primary path : {}", path, filePrimary.getAbsolutePath()); + Files.createSymbolicLink(Paths.get(path), Paths.get(filePrimary.getAbsolutePath())); + } + } + if (objMeta.isReplicaExist()) { + if (fileReplica == null) { + clientReplica.putFlush(); + // OSDClientManager.getInstance().returnOSDClient(clientReplica); + clientReplica = null; + } else { + fosReplica.flush(); + fosReplica.close(); + + if (fileReplica.exists()) { + File tempFile = new File(fileReplica.getAbsolutePath()); + logger.debug("fileReplica is already exists : {}", fileReplica.getAbsolutePath()); + retryRenameTo(tempFile, trashReplica); + } + // setAttributeFileReplication(tmpFileReplica, GWConstants.FILE_ATTRIBUTE_REPLICATION_REPLICA, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + retryRenameTo(tmpFileReplica, fileReplica); + if (isReplicaCache) { + String path = KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId); + com.google.common.io.Files.createParentDirs(new File(path)); + logger.debug("path : {}, primary path : {}", path, fileReplica.getAbsolutePath()); + Files.createSymbolicLink(Paths.get(path), Paths.get(fileReplica.getAbsolutePath())); + } + } + } + } else { + File file = null; + File tmpFile = null; + File trashFile = null; + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + File link = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + com.google.common.io.Files.createParentDirs(file); + com.google.common.io.Files.createParentDirs(tmpFile); + com.google.common.io.Files.createParentDirs(link); + fosPrimary = new FileOutputStream(tmpFile, false); + isPrimaryCache = true; + } else { + file = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFile = new File(KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + trashFile = new File(KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + com.google.common.io.Files.createParentDirs(file); + com.google.common.io.Files.createParentDirs(tmpFile); + fosPrimary = new FileOutputStream(tmpFile, false); + } + + } else { + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (clientPrimary == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + clientPrimary = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + + logger.info("osd - {},{}", clientPrimary.getSocket().getRemoteSocketAddress().toString(), clientPrimary.getSocket().getLocalPort()); + clientPrimary.putInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + versionId, + length, + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + GWConstants.EMPTY_STRING,//objMeta.getReplicaDisk().getId(), + GWConstants.EMPTY_STRING, + GWConfig.getInstance().getPerformanceMode()); + } + + while ((readLength = azuParameter.getInputStream().read(buffer, 0, GWConstants.BUFSIZE)) != -1) { + totalReads += readLength; + if (file == null) { + clientPrimary.put(buffer, 0, readLength); + } else { + fosPrimary.write(buffer, 0, readLength); + } + + md5er.update(buffer, 0, readLength); + + if (totalReads >= length) { + break; + } + } + logger.info("total read : {}", totalReads); + + if (file == null) { + clientPrimary.putFlush(); + // OSDClientManager.getInstance().returnOSDClient(clientPrimary); + clientPrimary = null; + } else { + fosPrimary.flush(); + fosPrimary.close(); + + if (file.exists()) { + File tempFile = new File(file.getAbsolutePath()); + logger.debug("file is already exists : {}", file.getAbsolutePath()); + retryRenameTo(tempFile, trashFile); + } + + // setAttributeFileReplication(tmpFile, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + + retryRenameTo(tmpFile, file); + if (isPrimaryCache) { + String path = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + Files.createSymbolicLink(Paths.get(path), Paths.get(file.getAbsolutePath())); + } + } + } + + byte[] digest = md5er.digest(); + String eTag = base64().encode(digest); + + s3Object.setEtag(eTag); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(totalReads); + s3Object.setVersionId(versionId); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); + + calSize = putSize - existFileSize; + if (GWConfig.getInstance().isNoOption()) { + updateBucketUsed(objMeta.getBucket(), calSize); + } + } catch (NoSuchAlgorithmException | IOException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } catch (ResourceNotFoundException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.NO_SUCH_KEY, azuParameter); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + if (objMeta.getReplicaCount() > 1) { + if (fosPrimary != null) { + try { + fosPrimary.close(); + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + if (fosReplica != null) { + try { + fosReplica.close(); + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + } + try { + if (clientPrimary != null) { + // OSDClientManager.getInstance().returnOSDClient(clientPrimary); + } + if (clientReplica != null) { + // OSDClientManager.getInstance().returnOSDClient(clientReplica); + } + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } + return s3Object; + } + + public boolean deleteObject() throws AzuException { + OSDClient client = null; + try { + if (objMeta.getReplicaCount() > 1) { + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + deleteObjectLocal(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId()); + } else { + client = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + // client = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + client.delete(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId()); + // OSDClientManager.getInstance().returnOSDClient(client); + client = null; + } + + if (objMeta.isReplicaExist()) { + if (GWUtils.getLocalIP().equals(objMeta.getReplicaDisk().getOsdIp())) { + deleteObjectLocal(objMeta.getReplicaDisk().getPath(), objMeta.getObjId()); + } else { + client = new OSDClient(objMeta.getReplicaDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + // client = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + client.delete(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId()); + // OSDClientManager.getInstance().returnOSDClient(client); + client = null; + } + } + + } else { + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + deleteObjectLocal(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId()); + } else { + client = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + // client = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + client.delete(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId()); + // OSDClientManager.getInstance().returnOSDClient(client); + client = null; + } + } + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + if (client != null) { + try { + // OSDClientManager.getInstance().returnOSDClient(client); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + } + logger.info(AzuConstants.LOG_S3OBJECT_OPERATION_DELETE, objMeta.getBucket(), objMeta.getPath()); + return true; + } + + private void deleteObjectLocal(String path, String objId) throws IOException, AzuException { + File file = null; + File trashFile = null; + if (GWConfig.getInstance().isCacheDiskpath()) { + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(path, objId, versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(path, objId, versionId))); + } else { + file = new File(KsanUtils.makeObjPath(path, objId, versionId)); + trashFile = new File(KsanUtils.makeTrashPath(path, objId, versionId)); + } + + updateBucketUsed(objMeta.getBucket(), file.length() * objMeta.getReplicaCount() * -1); + if (file.exists()) { + retryRenameTo(file, trashFile); + if (GWConfig.getInstance().isCacheDiskpath()) { + File link = new File(KsanUtils.makeObjPath(path, objId, versionId)); + link.delete(); + } + } + } + + public S3Object uploadBlock(String blockId, long length) throws AzuException { + S3Object s3Object = new S3Object(); + MessageDigest md5er = null; + byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; + int readLength = 0; + long totalReads = 0L; + File filePrimary = null; + File fileReplica = null; + FileOutputStream fosPrimary = null; + FileOutputStream fosReplica = null; + OSDClient clientPrimary = null; + OSDClient clientReplica = null; + + try { + md5er = MessageDigest.getInstance(GWConstants.MD5, new OpenSSL4JProvider()); + if (objMeta.getReplicaCount() > 1) { + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + filePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), blockId))); + } else { + filePrimary = new File(KsanUtils.makeTempPartPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), blockId)); + } + com.google.common.io.Files.createParentDirs(filePrimary); + fosPrimary = new FileOutputStream(filePrimary, false); + } else { + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (clientPrimary == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + clientPrimary = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientPrimary.getSocket().getRemoteSocketAddress().toString(), clientPrimary.getSocket().getLocalPort()); + clientPrimary.partInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + blockId, + length, + GWConstants.EMPTY_STRING); + } + + if (objMeta.isReplicaExist()) { + if (GWUtils.getLocalIP().equals(objMeta.getReplicaDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + fileReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), blockId))); + } else { + fileReplica = new File(KsanUtils.makeTempPartPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), blockId)); + } + com.google.common.io.Files.createParentDirs(fileReplica); + fosReplica = new FileOutputStream(fileReplica, false); + } else { + // clientReplica = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // if (clientReplica == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getReplicaDisk().getOsdIp()); + // clientReplica = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // } + clientReplica = new OSDClient(objMeta.getReplicaDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientReplica.getSocket().getRemoteSocketAddress().toString(), clientReplica.getSocket().getLocalPort()); + clientReplica.partInit(objMeta.getReplicaDisk().getPath(), + objMeta.getObjId(), + blockId, + length, + GWConstants.EMPTY_STRING); + } + } + + while ((readLength = azuParameter.getInputStream().read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + totalReads += readLength; + if (filePrimary == null) { + clientPrimary.put(buffer, 0, readLength); + } else { + fosPrimary.write(buffer, 0, readLength); + } + + if (objMeta.isReplicaExist()) { + if (fileReplica == null) { + clientReplica.put(buffer, 0, readLength); + } else { + fosReplica.write(buffer, 0, readLength); + } + } + + md5er.update(buffer, 0, readLength); + if (totalReads >= length) { + break; + } + } + + if (filePrimary == null) { + clientPrimary.putFlush(); + clientPrimary = null; + } else { + fosPrimary.flush(); + fosPrimary.close(); + } + + if (objMeta.isReplicaExist()) { + if (fileReplica == null) { + clientReplica.putFlush(); + clientReplica = null; + } else { + fosReplica.flush(); + fosReplica.close(); + } + } + } else { + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + filePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), blockId))); + } else { + filePrimary = new File(KsanUtils.makeTempPartPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), blockId)); + } + com.google.common.io.Files.createParentDirs(filePrimary); + fosPrimary = new FileOutputStream(filePrimary, false); + } else { + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (clientPrimary == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + clientPrimary = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientPrimary.getSocket().getRemoteSocketAddress().toString(), clientPrimary.getSocket().getLocalPort()); + clientPrimary.partInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + blockId, + length, + GWConstants.EMPTY_STRING); + } + + while ((readLength = azuParameter.getInputStream().read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + totalReads += readLength; + if (filePrimary == null) { + clientPrimary.put(buffer, 0, readLength); + } else { + fosPrimary.write(buffer, 0, readLength); + } + + md5er.update(buffer, 0, readLength); + if (totalReads >= length) { + break; + } + } + + if (filePrimary == null) { + clientPrimary.putFlush(); + clientPrimary = null; + } else { + fosPrimary.flush(); + fosPrimary.close(); + } + } + + logger.debug("Total read : {}", totalReads); + + byte[] digest = md5er.digest(); + String eTag = base64().encode(digest); + + s3Object.setEtag(eTag); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(length); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + return s3Object; + } + + public S3Object completeBlockList(Collection blockList) throws AzuException { + S3Object s3Object = new S3Object(); + MessageDigest md5er = null; + byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; + File file = null; + File tmpFile = null; + File trashFile = null; + File block = null; + int readLength = 0; + long totalLength = 0L; + long existFileSize = 0L; + long putSize = 0L; + long calSize = 0L; + boolean isCacheDiskpath = false; + + if (GWConfig.getInstance().isCacheDiskpath()) { + isCacheDiskpath = true; + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempCompleteMultipartPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + } else { + file = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFile = new File(KsanUtils.makeTempCompleteMultipartPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + trashFile = new File(KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + } + + try (FileOutputStream tmpOut = new FileOutputStream(tmpFile)) { + com.google.common.io.Files.createParentDirs(file); + com.google.common.io.Files.createParentDirs(tmpFile); + md5er = MessageDigest.getInstance(GWConstants.MD5, new OpenSSL4JProvider()); + for (String blockId: blockList) { + if (GWConfig.getInstance().isCacheDiskpath()) { + block = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), blockId))); + } else { + block = new File(KsanUtils.makeTempPartPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), blockId)); + } + + FileInputStream fis = new FileInputStream(block); + while ((readLength = fis.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + totalLength += readLength; + tmpOut.write(buffer, 0, readLength); + md5er.update(buffer, 0, readLength); + } + fis.close(); + + block.delete(); + } + + if (file.exists()) { + File tempFile = new File(file.getAbsolutePath()); + retryRenameTo(tempFile, trashFile); + } + // if (objMeta.getReplicaDisk() != null && !Strings.isNullOrEmpty(objMeta.getReplicaDisk().getId())) { + // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId()); + // } else { + // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + // } + retryRenameTo(tmpFile, file); + if (isCacheDiskpath) { + String filePath = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + Files.createSymbolicLink(Paths.get(filePath), Paths.get(file.getAbsolutePath())); + } + + byte[] digest = md5er.digest(); + String eTag = base64().encode(digest); + s3Object.setEtag(eTag); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(totalLength); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); + } catch (NoSuchAlgorithmException | IOException e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + + return s3Object; + } + + private void setObjManager() throws Exception { + objManager = ObjManagerHelper.getInstance().getObjManager(); + } + + private void releaseObjManager() throws Exception { + ObjManagerHelper.getInstance().returnObjManager(objManager); + } + + private void updateBucketUsed(String bucketName, long size) throws AzuException { + try { + setObjManager(); + objManager.updateBucketUsed(bucketName, size); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } finally { + try { + releaseObjManager(); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new AzuException(AzuErrorCode.SERVER_ERROR, azuParameter); + } + } + } + + private void retryRenameTo(File srcFile, File destFile) throws IOException { + if (srcFile.exists()) { + for (int i = 0; i < GWConstants.RETRY_COUNT; i++) { + if (srcFile.renameTo(destFile)) { + return; + } + try { + Thread.sleep(100); + } catch (InterruptedException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + } + logger.error(GWConstants.LOG_S3OBJECT_OPERATION_FAILED_FILE_RENAME, srcFile.getAbsolutePath(), destFile.getAbsolutePath()); + } + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/object/S3Encryption.java b/core/src/com/pspace/ifs/ksan/gw/object/S3Encryption.java index f4aa885e..261923c5 100644 --- a/core/src/com/pspace/ifs/ksan/gw/object/S3Encryption.java +++ b/core/src/com/pspace/ifs/ksan/gw/object/S3Encryption.java @@ -29,6 +29,15 @@ public S3Encryption(S3Metadata s3Metadata, S3Parameter s3Parameter) { this.algorithm = s3Metadata.getServersideEncryption(); } + public S3Encryption(String serverSideEncryption, String customerAlgorithm, String customerKey, String customerKeyMD5, S3Parameter s3Parameter) { + this.s3Parameter = s3Parameter; + this.customerAlgorithm = customerAlgorithm; + this.customerKey = customerKey; + this.customerKeyMD5 = customerKeyMD5; + this.algorithm = serverSideEncryption; + } + + protected String makeMD5(String str) { String md5 = GWConstants.EMPTY_STRING; diff --git a/core/src/com/pspace/ifs/ksan/gw/object/S3ObjectOperation.java b/core/src/com/pspace/ifs/ksan/gw/object/S3ObjectOperation.java index 0ce60e15..9145f97c 100644 --- a/core/src/com/pspace/ifs/ksan/gw/object/S3ObjectOperation.java +++ b/core/src/com/pspace/ifs/ksan/gw/object/S3ObjectOperation.java @@ -19,7 +19,9 @@ import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; +import java.io.RandomAccessFile; import java.nio.ByteBuffer; +import java.nio.channels.FileChannel; import java.nio.charset.Charset; import java.nio.file.Files; import java.nio.file.Path; @@ -33,12 +35,14 @@ import java.util.Iterator; import java.util.Map; import java.util.SortedMap; -import java.util.UUID; +import java.util.List; +import java.util.ArrayList; import com.google.common.base.Strings; import com.google.common.primitives.Longs; import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.libs.identity.S3Metadata; import com.pspace.ifs.ksan.gw.identity.S3Parameter; import com.pspace.ifs.ksan.libs.multipart.Part; @@ -54,7 +58,15 @@ import com.pspace.ifs.ksan.libs.DiskManager; import com.pspace.ifs.ksan.libs.OSDClient; import com.pspace.ifs.ksan.libs.data.OsdData; +import com.pspace.ifs.ksan.libs.KsanUtils; +import com.pspace.ifs.ksan.libs.disk.Disk; +import com.pspace.ifs.ksan.libs.disk.DiskPool; +import com.pspace.ifs.ksan.libs.disk.Server; +import com.pspace.ifs.ksan.libs.data.ECPart; +import de.sfuhrm.openssl4j.OpenSSL4JProvider; + +import org.apache.commons.io.FileUtils; import org.apache.commons.crypto.stream.CtrCryptoInputStream; import org.apache.commons.crypto.stream.CtrCryptoOutputStream; import org.slf4j.Logger; @@ -96,6 +108,24 @@ public void getObject(S3Range s3Range) throws Exception { long fileSize = objMeta.getSize(); String key = Strings.isNullOrEmpty(s3Encryption.getCustomerKey()) ? GWConstants.EMPTY_STRING : s3Encryption.getCustomerKey(); + if (GWConfig.getInstance().isNoReplica() || GWConfig.getInstance().isNoIO()) { + byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; + int sendSize = 0; + long remainingLength = fileSize; + + while (remainingLength > 0) { + if (remainingLength > GWConstants.MAXBUFSIZE) { + sendSize = GWConstants.MAXBUFSIZE; + } else { + sendSize = (int)remainingLength; + } + s3Parameter.getResponse().getOutputStream().write(buffer, 0, sendSize); + remainingLength -= sendSize; + } + + return; + } + if (s3Range != null && s3Range.getListRange().size() > 0) { fileSize = 0L; for (S3Range.Range range : s3Range.getListRange()) { @@ -107,7 +137,188 @@ public void getObject(S3Range s3Range) throws Exception { fileSize += range.getLength(); } } - + + // check EC exists + // File ecFile = new File(KsanUtils.makeECPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + // logger.debug("ecfile : {}", ecFile.getAbsolutePath()); + // if (ecFile.exists()) { + // List ecList = new ArrayList(); + // for (DiskPool pool : DiskManager.getInstance().getDiskPoolList()) { + // for (Server server : pool.getServerList()) { + // for (Disk disk : server.getDiskList()) { + // ECPart ecPart = new ECPart(server.getIp(), disk.getPath(), false); + // ecList.add(ecPart); + // } + // } + // } + // int numberOfCodingChunks = DiskManager.getInstance().getECM(objMeta.getPrimaryDisk().getId()); + // int numberOfDataChunks = DiskManager.getInstance().getECK(objMeta.getPrimaryDisk().getId()); + // logger.debug("numberOfCodingChunks : {}, numberOfDataChunks : {}", numberOfCodingChunks, numberOfDataChunks); + // int getECPartCount = 0; + // for (ECPart ecPart : ecList) { + // String newECPartPath = ecFile.getAbsolutePath() + Constants.POINT + Integer.toString(getECPartCount); + // File newECPartFile = new File(newECPartPath); + // if (ecPart.getServerIP().equals(GWUtils.getLocalIP())) { + // // if local disk, move file + // if (ecPart.getDiskPath().equals(objMeta.getPrimaryDisk().getPath())) { + // FileUtils.copyFile(ecFile, newECPartFile); + // ecPart.setProcessed(true); + // } else { + // File sourceECPartFile = new File(KsanUtils.makeECPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + // if (sourceECPartFile.exists()) { + // FileUtils.copyFile(sourceECPartFile, newECPartFile); + // ecPart.setProcessed(true); + // } else { + // logger.info("ec part does not exist. {}", sourceECPartFile.getAbsolutePath()); + // } + // } + // } else { + // try (FileOutputStream fos = new FileOutputStream(newECPartFile)) { + // String getPath = KsanUtils.makeECPath(ecPart.getDiskPath(), objMeta.getObjId(), versionId); + // OSDClient ecClient = new OSDClient(ecPart.getServerIP(), (int)GWConfig.getInstance().getOsdPort()); + // logger.debug("get ec part file : {}, to : {}, {}", getPath, ecPart.getServerIP(), ecPart.getDiskPath()); + // ecClient.getECPartInit(getPath, fos); + // ecClient.getECPart(); + // ecPart.setProcessed(true); + // } catch (IOException e) { + // PrintStack.logging(logger, e); + // } + // } + // getECPartCount++; + // } + // // zunfec + // String ecAllFilePath = KsanUtils.makeECDecodePath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + // String command = Constants.ZUNFEC + ecAllFilePath; + // getECPartCount = 0; + // for (ECPart ecPart : ecList) { + // String ecPartPath = ecFile.getAbsolutePath() + Constants.POINT + Integer.toString(getECPartCount); + // if (ecPart.isProcessed()) { + // command += Constants.SPACE + ecPartPath; + // getECPartCount++; + // } + // } + // logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_ZUNFEC_COMMAND, command); + // Process p = Runtime.getRuntime().exec(command); + // try { + // int exitCode = p.waitFor(); + // p.destroy(); + // logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_ZUNFEC_DECODE_EXIT_VALUE, exitCode); + // } catch (InterruptedException e) { + // logger.error(e.getMessage()); + // } + // File ecAllFile = new File(ecAllFilePath); + // if (ecAllFile.exists()) { + // logger.info("zunfec result : {}, {}", ecAllFile.getAbsolutePath(), ecAllFile.length()); + // byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; + // CtrCryptoInputStream encryptIS = null; + + // try (FileInputStream fis = new FileInputStream(ecAllFile); OutputStream os = s3Parameter.getResponse().getOutputStream()) { + // long remaingLength = 0L; + // int readLength = 0; + // int readBytes; + + // if (Strings.isNullOrEmpty(sourceRange)) { + // if (!Strings.isNullOrEmpty(key)) { + // encryptIS = GWUtils.initCtrDecrypt(fis, key); + // while ((readLength = encryptIS.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + // actualSize += readLength; + // os.write(buffer, 0, readLength); + // logger.debug("read length : {}", readLength); + // } + // } else { + // remaingLength = ecAllFile.length(); + // while (remaingLength > 0) { + // readBytes = 0; + // if (remaingLength < GWConstants.MAXBUFSIZE) { + // readBytes = (int)remaingLength; + // } else { + // readBytes = GWConstants.MAXBUFSIZE; + // } + + // if (remaingLength >= GWConstants.MAXBUFSIZE) { + // readLength = GWConstants.MAXBUFSIZE; + // } else { + // readLength = (int)remaingLength; + // } + // readLength = fis.read(buffer, 0, readBytes); + // logger.debug("read length : {}", readLength); + // actualSize += readLength; + // os.write(buffer, 0, readLength); + // remaingLength -= readLength; + // } + // } + // } else { + // String[] ranges = sourceRange.split(GWConstants.SLASH); + // for (String range : ranges) { + // String[] rangeParts = range.split(GWConstants.COMMA); + // long offset = Longs.tryParse(rangeParts[0]); + // long length = Longs.tryParse(rangeParts[1]); + // logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_RANGE, offset, length); + + // remaingLength = length; + + // if (!Strings.isNullOrEmpty(key)) { + // long skipOffset = 0; + // encryptIS = GWUtils.initCtrDecrypt(fis, key); + + // if (offset > 0) { + // long skip = encryptIS.skip(offset); + // logger.debug("skip : {}", skip); + // } + // while (remaingLength > 0) { + // readBytes = 0; + // if (remaingLength < GWConstants.MAXBUFSIZE) { + // readBytes = (int)remaingLength; + // } else { + // readBytes = GWConstants.MAXBUFSIZE; + // } + + // if ((readLength = encryptIS.read(buffer, 0, readBytes)) != -1) { + // skipOffset += readLength; + // logger.debug("read {} bytes", readLength); + // actualSize += readLength; + // os.write(buffer, 0, readLength); + // remaingLength -= readLength; + // } else { + // break; + // } + // } + // } else { + // if (offset > 0) { + // fis.skip(offset); + // } + // while (remaingLength > 0) { + // readBytes = 0; + // if (remaingLength < GWConstants.MAXBUFSIZE) { + // readBytes = (int)remaingLength; + // } else { + // readBytes = GWConstants.MAXBUFSIZE; + // } + + // readLength = fis.read(buffer, 0, readBytes); + + // actualSize += readLength; + // os.write(buffer, 0, readLength); + // remaingLength -= readLength; + // } + // } + // } + // } + // os.flush(); + // } catch (IOException e) { + // PrintStack.logging(logger, e); + // throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + // } + + // if (encryptIS != null) { + // encryptIS.close(); + // } + + // s3Parameter.addResponseSize(actualSize); + // return; + // } + // } + try { if (objMeta.getReplicaCount() > 1) { logger.debug("bucket : {}, object : {}", objMeta.getBucket(), objMeta.getPath()); @@ -195,7 +406,7 @@ public void getObject(S3Range s3Range) throws Exception { key); actualSize = client.get(); // OSDClientManager.getInstance().returnOSDClient(client); - client = null; + // client = null; s3Parameter.addResponseSize(actualSize); } catch (Exception e) { PrintStack.logging(logger, e); @@ -216,34 +427,18 @@ public void getObject(S3Range s3Range) throws Exception { private long getObjectLocal(OutputStream outputStream, String path, String objId, String sourceRange, String key) throws IOException, GWException, InvalidKeyException, NoSuchAlgorithmException { byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; - File ecFile = new File(makeECPath(path, objId, versionId)); - File file = null; + File file = new File(KsanUtils.makeObjPath(path, objId, versionId)); CtrCryptoInputStream encryptIS = null; - if (ecFile.exists()) { - logger.info(GWConstants.LOG_S3OBJECT_OPERATION_ZUNFEC_DECODE, ecFile.getName()); - String command = GWConstants.ZUNFEC - + makeECDecodePath(path, objId, versionId) - + GWConstants.SPACE + ecFile + GWConstants.ZFEC_0 - + GWConstants.SPACE + ecFile + GWConstants.ZFEC_1 - + GWConstants.SPACE + ecFile + GWConstants.ZFEC_2 - + GWConstants.SPACE + ecFile + GWConstants.ZFEC_3; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_ZUNFEC_COMMAND, command); - Process p = Runtime.getRuntime().exec(command); - try { - int exitCode = p.waitFor(); - p.destroy(); - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_ZUNFEC_DECODE_EXIT_VALUE, exitCode); - } catch (InterruptedException e) { - logger.error(e.getMessage()); - } + if (GWConfig.getInstance().isNoIO()) { + outputStream.write(buffer, 0, 100 * 1000); + outputStream.flush(); + outputStream.close(); - file = new File(makeECDecodePath(path, objId, versionId)); - } else { - file = new File(makeObjPath(path, objId, versionId)); - } + return 100*1000; + } - logger.info("obj path : {}", file.getAbsolutePath()); + // logger.info("obj path : {}", file.getAbsolutePath()); long actualSize = 0L; try (FileInputStream fis = new FileInputStream(file)) { long remaingLength = 0L; @@ -257,7 +452,6 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId actualSize += readLength; outputStream.write(buffer, 0, readLength); logger.debug("read length : {}", readLength); - // s3Parameter.addResponseSize(readLength); } } else { remaingLength = file.length(); @@ -278,7 +472,6 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId actualSize += readLength; outputStream.write(buffer, 0, readLength); - // s3Parameter.addResponseSize(readLength); remaingLength -= readLength; } } @@ -292,7 +485,6 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId remaingLength = length; - if (!Strings.isNullOrEmpty(key)) { long skipOffset = 0; encryptIS = GWUtils.initCtrDecrypt(fis, key); @@ -314,7 +506,6 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId logger.debug("read {} bytes", readLength); actualSize += readLength; outputStream.write(buffer, 0, readLength); - // s3Parameter.addResponseSize(readLength); remaingLength -= readLength; } else { break; @@ -336,12 +527,14 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId actualSize += readLength; outputStream.write(buffer, 0, readLength); - // s3Parameter.addResponseSize(readLength); remaingLength -= readLength; } } } } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } if (encryptIS != null) { @@ -350,41 +543,124 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId outputStream.flush(); outputStream.close(); - if (ecFile.exists()) { - file.delete(); - } - return actualSize; } private long getObjectLocal(OutputStream outputStream, String path, String objId, String versionId, String sourceRange, String key) throws IOException, GWException, InvalidKeyException, NoSuchAlgorithmException { byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; - File ecFile = new File(makeECPath(path, objId, versionId)); - File file = null; + File file = new File(KsanUtils.makeObjPath(path, objId, versionId));; CtrCryptoInputStream encryptIS = null; - if (ecFile.exists()) { - logger.info(GWConstants.LOG_S3OBJECT_OPERATION_ZUNFEC_DECODE, ecFile.getName()); - String command = GWConstants.ZUNFEC - + makeECDecodePath(path, objId, versionId) - + GWConstants.SPACE + ecFile + GWConstants.ZFEC_0 - + GWConstants.SPACE + ecFile + GWConstants.ZFEC_1 - + GWConstants.SPACE + ecFile + GWConstants.ZFEC_2 - + GWConstants.SPACE + ecFile + GWConstants.ZFEC_3; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_ZUNFEC_COMMAND, command); - Process p = Runtime.getRuntime().exec(command); - try { - int exitCode = p.waitFor(); - p.destroy(); - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_ZUNFEC_DECODE_EXIT_VALUE, exitCode); - } catch (InterruptedException e) { - logger.error(e.getMessage()); + long actualSize = 0L; + try (FileInputStream fis = new FileInputStream(file)) { + long remaingLength = 0L; + int readLength = 0; + int readBytes; + + if (Strings.isNullOrEmpty(sourceRange)) { + if (!Strings.isNullOrEmpty(key)) { + encryptIS = GWUtils.initCtrDecrypt(fis, key); + while ((readLength = encryptIS.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + actualSize += readLength; + outputStream.write(buffer, 0, readLength); + } + } else { + remaingLength = file.length(); + while (remaingLength > 0) { + readBytes = 0; + if (remaingLength < GWConstants.MAXBUFSIZE) { + readBytes = (int)remaingLength; + } else { + readBytes = GWConstants.MAXBUFSIZE; + } + + if (remaingLength >= GWConstants.MAXBUFSIZE) { + readLength = GWConstants.MAXBUFSIZE; + } else { + readLength = (int)remaingLength; + } + readLength = fis.read(buffer, 0, readBytes); + + actualSize += readLength; + outputStream.write(buffer, 0, readLength); + remaingLength -= readLength; + } + } + } else { + String[] ranges = sourceRange.split(GWConstants.SLASH); + for (String range : ranges) { + String[] rangeParts = range.split(GWConstants.COMMA); + long offset = Longs.tryParse(rangeParts[0]); + long length = Longs.tryParse(rangeParts[1]); + logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_RANGE, offset, length); + + remaingLength = length; + + if (!Strings.isNullOrEmpty(key)) { + long skipOffset = 0; + encryptIS = GWUtils.initCtrDecrypt(fis, key); + + if (offset > 0) { + long skip = encryptIS.skip(offset); + logger.debug("skip : {}", skip); + } + while (remaingLength > 0) { + readBytes = 0; + if (remaingLength < GWConstants.MAXBUFSIZE) { + readBytes = (int)remaingLength; + } else { + readBytes = GWConstants.MAXBUFSIZE; + } + + if ((readLength = encryptIS.read(buffer, 0, readBytes)) != -1) { + skipOffset += readLength; + logger.debug("read {} bytes", readLength); + actualSize += readLength; + outputStream.write(buffer, 0, readLength); + remaingLength -= readLength; + } else { + break; + } + } + } else { + if (offset > 0) { + fis.skip(offset); + } + while (remaingLength > 0) { + readBytes = 0; + if (remaingLength < GWConstants.MAXBUFSIZE) { + readBytes = (int)remaingLength; + } else { + readBytes = GWConstants.MAXBUFSIZE; + } + + readLength = fis.read(buffer, 0, readBytes); + + actualSize += readLength; + outputStream.write(buffer, 0, readLength); + remaingLength -= readLength; + } + } + } } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } - file = new File(makeECDecodePath(path, objId, versionId)); - } else { - file = new File(makeObjPath(path, objId, versionId)); + if (encryptIS != null) { + encryptIS.close(); } + outputStream.flush(); + outputStream.close(); + + return actualSize; + } + + private long getObjectLocal(OutputStream outputStream, String path, String objId, String versionId, String sourceRange, MessageDigest md5er, String key) throws IOException, GWException, InvalidKeyException, NoSuchAlgorithmException { + byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; + File file = new File(KsanUtils.makeObjPath(path, objId, versionId)); + CtrCryptoInputStream encryptIS = null; long actualSize = 0L; try (FileInputStream fis = new FileInputStream(file)) { @@ -398,7 +674,7 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId while ((readLength = encryptIS.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { actualSize += readLength; outputStream.write(buffer, 0, readLength); - // s3Parameter.addResponseSize(readLength); + md5er.update(buffer, 0, readLength); } } else { remaingLength = file.length(); @@ -419,7 +695,7 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId actualSize += readLength; outputStream.write(buffer, 0, readLength); - // s3Parameter.addResponseSize(readLength); + md5er.update(buffer, 0, readLength); remaingLength -= readLength; } } @@ -432,7 +708,6 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_RANGE, offset, length); remaingLength = length; - if (!Strings.isNullOrEmpty(key)) { long skipOffset = 0; @@ -455,7 +730,7 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId logger.debug("read {} bytes", readLength); actualSize += readLength; outputStream.write(buffer, 0, readLength); - // s3Parameter.addResponseSize(readLength); + md5er.update(buffer, 0, readLength); remaingLength -= readLength; } else { break; @@ -477,12 +752,15 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId actualSize += readLength; outputStream.write(buffer, 0, readLength); - // s3Parameter.addResponseSize(readLength); + md5er.update(buffer, 0, readLength); remaingLength -= readLength; } } } } + } catch (IOException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } if (encryptIS != null) { @@ -491,26 +769,11 @@ private long getObjectLocal(OutputStream outputStream, String path, String objId outputStream.flush(); outputStream.close(); - if (ecFile.exists()) { - file.delete(); - } - return actualSize; } public S3Object putObject() throws GWException { S3Object s3Object = null; - String objectName = s3Parameter.getObjectName(); - - if (getDirectoryBlobSuffix(objectName) != null) { - s3Object = new S3Object(); - s3Object.setVersionId(GWConstants.VERSIONING_DISABLE_TAIL); - s3Object.setEtag(GWConstants.DIRECTORY_MD5); - s3Object.setLastModified(new Date()); - s3Object.setFileSize(0); - s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); - return s3Object; - } if (s3Encryption.isEncryptionEnabled()) { s3Object = putObjectEncryption(s3Meta.getContentLength(), s3Parameter.getInputStream()); @@ -525,8 +788,9 @@ public S3Object putObject() throws GWException { } private S3Object putObjectNormal(long length, InputStream is) throws GWException { + long putStart = System.currentTimeMillis(); + long opEnd = 0L; S3Object s3Object = new S3Object(); - File filePrimary = null; File tmpFilePrimary = null; FileOutputStream fosPrimary = null; @@ -543,7 +807,7 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException long calSize = 0L; try { - MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5); + MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5, new OpenSSL4JProvider()); byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; int readLength = 0; @@ -568,14 +832,14 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - filePrimary = new File(makeCachePath(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - tmpFilePrimary = new File(makeCachePath(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - trashPrimary = new File(makeCachePath(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + filePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFilePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + trashPrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); isPrimaryCache = true; } else { - filePrimary = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - tmpFilePrimary = new File(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - trashPrimary = new File(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + filePrimary = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFilePrimary = new File(KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + trashPrimary = new File(KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); } com.google.common.io.Files.createParentDirs(filePrimary); com.google.common.io.Files.createParentDirs(tmpFilePrimary); @@ -592,7 +856,7 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId(), GWConstants.EMPTY_STRING, GWConfig.getInstance().getPerformanceMode()); @@ -601,14 +865,14 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException if (objMeta.isReplicaExist()) { if (GWUtils.getLocalIP().equals(objMeta.getReplicaDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - fileReplica = new File(makeCachePath(makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); - tmpFileReplica = new File(makeCachePath(makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); - trashReplica = new File(makeCachePath(makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); + fileReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFileReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); + trashReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); isReplicaCache = true; } else { - fileReplica = new File(makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); - tmpFileReplica = new File(makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); - trashReplica = new File(makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + fileReplica = new File(KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFileReplica = new File(KsanUtils.makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + trashReplica = new File(KsanUtils.makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); } com.google.common.io.Files.createParentDirs(fileReplica); com.google.common.io.Files.createParentDirs(tmpFileReplica); @@ -625,8 +889,8 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRIBUTE_REPLICATION_REPLICA, - GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, + Constants.FILE_ATTRIBUTE_REPLICATION_REPLICA, + Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, GWConstants.EMPTY_STRING, GWConfig.getInstance().getPerformanceMode()); } @@ -670,15 +934,16 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException retryRenameTo(tempFile, trashPrimary); } + KsanUtils.setAttributeFileReplication(tmpFilePrimary, Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId()); // if (objMeta.getReplicaDisk() != null && !Strings.isNullOrEmpty(objMeta.getReplicaDisk().getId())) { // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId()); // } else { - // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); // } retryRenameTo(tmpFilePrimary, filePrimary); if (isPrimaryCache) { - String path = makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + String path = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); com.google.common.io.Files.createParentDirs(new File(path)); logger.debug("path : {}, primary path : {}", path, filePrimary.getAbsolutePath()); Files.createSymbolicLink(Paths.get(path), Paths.get(filePrimary.getAbsolutePath())); @@ -698,10 +963,11 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException logger.debug("fileReplica is already exists : {}", fileReplica.getAbsolutePath()); retryRenameTo(tempFile, trashReplica); } - // setAttributeFileReplication(tmpFileReplica, GWConstants.FILE_ATTRIBUTE_REPLICATION_REPLICA, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + KsanUtils.setAttributeFileReplication(tmpFileReplica, Constants.FILE_ATTRIBUTE_REPLICATION_REPLICA, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + retryRenameTo(tmpFileReplica, fileReplica); if (isReplicaCache) { - String path = makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId); + String path = KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId); com.google.common.io.Files.createParentDirs(new File(path)); logger.debug("path : {}, primary path : {}", path, fileReplica.getAbsolutePath()); Files.createSymbolicLink(Paths.get(path), Paths.get(fileReplica.getAbsolutePath())); @@ -714,24 +980,23 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException File trashFile = null; if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - file = new File(makeCachePath(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - tmpFile = new File(makeCachePath(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - trashFile = new File(makeCachePath(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - File link = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + File link = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); com.google.common.io.Files.createParentDirs(link); fosPrimary = new FileOutputStream(tmpFile, false); isPrimaryCache = true; } else { - file = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - tmpFile = new File(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - trashFile = new File(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + file = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFile = new File(KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + trashFile = new File(KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); fosPrimary = new FileOutputStream(tmpFile, false); } - } else { // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); // if (clientPrimary == null) { @@ -745,12 +1010,13 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.EMPTY_STRING,//objMeta.getReplicaDisk().getId(), GWConstants.EMPTY_STRING, GWConfig.getInstance().getPerformanceMode()); } - + // long opStart = System.currentTimeMillis(); + // logger.info("put pre : {}", opStart - putStart); while ((readLength = is.read(buffer, 0, GWConstants.BUFSIZE)) != -1) { totalReads += readLength; if (file == null) { @@ -765,7 +1031,9 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException break; } } - logger.info("total read : {}", totalReads); + opEnd = System.currentTimeMillis(); + // logger.info("put write : {}", opEnd - opStart); + // logger.info("total read : {}", totalReads); if (file == null) { clientPrimary.putFlush(); @@ -781,11 +1049,11 @@ private S3Object putObjectNormal(long length, InputStream is) throws GWException retryRenameTo(tempFile, trashFile); } - // setAttributeFileReplication(tmpFile, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + KsanUtils.setAttributeFileReplication(tmpFile, Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); retryRenameTo(tmpFile, file); if (isPrimaryCache) { - String path = makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + String path = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); Files.createSymbolicLink(Paths.get(path), Paths.get(file.getAbsolutePath())); } } @@ -798,19 +1066,19 @@ else if (GWConfig.getInstance().isNoReplica()) { File trashFile = null; if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - file = new File(makeCachePath(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - tmpFile = new File(makeCachePath(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - trashFile = new File(makeCachePath(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - File link = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + File link = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); com.google.common.io.Files.createParentDirs(link); fosPrimary = new FileOutputStream(tmpFile, false); isPrimaryCache = true; } else { - file = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - tmpFile = new File(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - trashFile = new File(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + file = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFile = new File(KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + trashFile = new File(KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); fosPrimary = new FileOutputStream(tmpFile, false); @@ -824,8 +1092,8 @@ else if (GWConfig.getInstance().isNoReplica()) { objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, - objMeta.getReplicaDisk().getId(), + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, //objMeta.getReplicaDisk().getId(), GWConstants.EMPTY_STRING, GWConfig.getInstance().getPerformanceMode()); } @@ -859,11 +1127,11 @@ else if (GWConfig.getInstance().isNoReplica()) { retryRenameTo(tempFile, trashFile); } - // setAttributeFileReplication(tmpFile, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + KsanUtils.setAttributeFileReplication(tmpFile, Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); retryRenameTo(tmpFile, file); if (isPrimaryCache) { - String path = makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + String path = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); Files.createSymbolicLink(Paths.get(path), Paths.get(file.getAbsolutePath())); } } @@ -885,7 +1153,7 @@ else if (GWConfig.getInstance().isNoDisk()) { objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId(), GWConstants.EMPTY_STRING, GWConfig.getInstance().getPerformanceMode()); @@ -898,8 +1166,8 @@ else if (GWConfig.getInstance().isNoDisk()) { objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRIBUTE_REPLICATION_REPLICA, - GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, + Constants.FILE_ATTRIBUTE_REPLICATION_REPLICA, + Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, GWConstants.EMPTY_STRING, GWConfig.getInstance().getPerformanceMode()); } @@ -932,13 +1200,15 @@ else if (GWConfig.getInstance().isNoDisk()) { // OSDClientManager.getInstance().returnOSDClient(clientReplica); clientReplica = null; } - } else { - logger.error(GWConstants.LOG_S3OBJECT_OPERATION_OPTION_NO_CASE, GWConfig.getInstance().getPerformanceMode()); - throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); - } + } + // else { + // logger.error(GWConstants.LOG_S3OBJECT_OPERATION_OPTION_NO_CASE, GWConfig.getInstance().getPerformanceMode()); + // throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + // } byte[] digest = md5er.digest(); String eTag = base16().lowerCase().encode(digest); + // String eTag = GWConstants.DIRECTORY_MD5; s3Object.setEtag(eTag); s3Object.setLastModified(new Date()); @@ -989,6 +1259,9 @@ else if (GWConfig.getInstance().isNoDisk()) { PrintStack.logging(logger, e); } } + + // long putEnd = System.currentTimeMillis(); + // logger.info("put after : {}", putEnd - opEnd); return s3Object; } @@ -1013,7 +1286,7 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep long calSize = 0L; try { - MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5); + MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5, new OpenSSL4JProvider()); byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; int readLength = 0; @@ -1038,10 +1311,10 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep } if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - filePrimary = new File(makeCachePath(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - tmpFilePrimary = new File(makeCachePath(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - trashPrimary = new File(makeCachePath(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - File file = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + filePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFilePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + trashPrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + File file = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(filePrimary); com.google.common.io.Files.createParentDirs(tmpFilePrimary); @@ -1050,9 +1323,9 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep fosPrimary = new FileOutputStream(tmpFilePrimary, false); isPrimaryCache = true; } else { - filePrimary = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - tmpFilePrimary = new File(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - trashPrimary = new File(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + filePrimary = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFilePrimary = new File(KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + trashPrimary = new File(KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(filePrimary); com.google.common.io.Files.createParentDirs(tmpFilePrimary); fosPrimary = new FileOutputStream(tmpFilePrimary, false); @@ -1065,7 +1338,7 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId(), s3Encryption.getCustomerKey(), GWConfig.getInstance().getPerformanceMode()); @@ -1074,10 +1347,10 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep if (objMeta.isReplicaExist()) { if (GWUtils.getLocalIP().equals(objMeta.getReplicaDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - fileReplica = new File(makeCachePath(makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); - tmpFileReplica = new File(makeCachePath(makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); - trashReplica = new File(makeCachePath(makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); - File file = new File(makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + fileReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFileReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); + trashReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId))); + File file = new File(KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(fileReplica); com.google.common.io.Files.createParentDirs(tmpFileReplica); @@ -1086,9 +1359,9 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep fosReplica = new FileOutputStream(tmpFileReplica, false); isReplicaCache = true; } else { - fileReplica = new File(makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); - tmpFileReplica = new File(makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); - trashReplica = new File(makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + fileReplica = new File(KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFileReplica = new File(KsanUtils.makeTempPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); + trashReplica = new File(KsanUtils.makeTrashPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(fileReplica); com.google.common.io.Files.createParentDirs(tmpFileReplica); fosReplica = new FileOutputStream(tmpFileReplica, false); @@ -1101,8 +1374,8 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRIBUTE_REPLICATION_REPLICA, - GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, + Constants.FILE_ATTRIBUTE_REPLICATION_REPLICA, + Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, s3Encryption.getCustomerKey(), GWConfig.getInstance().getPerformanceMode()); } @@ -1148,9 +1421,10 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep // } else { // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); // } + KsanUtils.setAttributeFileReplication(tmpFilePrimary, Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId()); retryRenameTo(tmpFilePrimary, filePrimary); if (isPrimaryCache) { - String path = makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + String path = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); Files.createSymbolicLink(Paths.get(path), Paths.get(filePrimary.getAbsolutePath())); } } @@ -1167,10 +1441,10 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep File tempFile = new File(fileReplica.getAbsolutePath()); retryRenameTo(tempFile, trashReplica); } - // setAttributeFileReplication(tmpFileReplica, GWConstants.FILE_ATTRIBUTE_REPLICATION_REPLICA, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + KsanUtils.setAttributeFileReplication(tmpFileReplica, Constants.FILE_ATTRIBUTE_REPLICATION_REPLICA, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); retryRenameTo(tmpFileReplica, fileReplica); if (isReplicaCache) { - String path = makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId); + String path = KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId); Files.createSymbolicLink(Paths.get(path), Paths.get(fileReplica.getAbsolutePath())); } } @@ -1181,19 +1455,19 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep File trashFile = null; if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - file = new File(makeCachePath(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - tmpFile = new File(makeCachePath(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - trashFile = new File(makeCachePath(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - File link = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + File link = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); com.google.common.io.Files.createParentDirs(link); fosPrimary = new FileOutputStream(tmpFile, false); isPrimaryCache = true; } else { - file = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - tmpFile = new File(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - trashFile = new File(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + file = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFile = new File(KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + trashFile = new File(KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); fosPrimary = new FileOutputStream(tmpFile, false); @@ -1206,9 +1480,9 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, - objMeta.getReplicaDisk().getId(), - s3Encryption.getCustomerKey(), + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL,//objMeta.getReplicaDisk().getId(), + GWConstants.EMPTY_STRING, GWConfig.getInstance().getPerformanceMode()); } @@ -1241,11 +1515,11 @@ private S3Object putObjectEncryption(long length, InputStream is) throws GWExcep retryRenameTo(tempFile, trashFile); } - // setAttributeFileReplication(tmpFile, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + KsanUtils.setAttributeFileReplication(tmpFile, Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); retryRenameTo(tmpFile, file); if (isPrimaryCache) { - String path = makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + String path = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); Files.createSymbolicLink(Paths.get(path), Paths.get(file.getAbsolutePath())); } } @@ -1258,19 +1532,19 @@ else if (GWConfig.getInstance().isNoReplica()) { File trashFile = null; if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - file = new File(makeCachePath(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - tmpFile = new File(makeCachePath(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - trashFile = new File(makeCachePath(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); - File link = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId))); + File link = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); com.google.common.io.Files.createParentDirs(link); fosPrimary = new FileOutputStream(tmpFile, false); isPrimaryCache = true; } else { - file = new File(makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - tmpFile = new File(makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); - trashFile = new File(makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + file = new File(KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + tmpFile = new File(KsanUtils.makeTempPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + trashFile = new File(KsanUtils.makeTrashPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); fosPrimary = new FileOutputStream(tmpFile, false); @@ -1283,8 +1557,8 @@ else if (GWConfig.getInstance().isNoReplica()) { objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, - objMeta.getReplicaDisk().getId(), + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, //objMeta.getReplicaDisk().getId(), s3Encryption.getCustomerKey(), GWConfig.getInstance().getPerformanceMode()); } @@ -1317,11 +1591,11 @@ else if (GWConfig.getInstance().isNoReplica()) { retryRenameTo(tempFile, trashFile); } - // setAttributeFileReplication(tmpFile, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + KsanUtils.setAttributeFileReplication(tmpFile, Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); retryRenameTo(tmpFile, file); if (isPrimaryCache) { - String path = makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + String path = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); Files.createSymbolicLink(Paths.get(path), Paths.get(file.getAbsolutePath())); } } @@ -1342,7 +1616,7 @@ else if (GWConfig.getInstance().isNoDisk()) { objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, + Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId(), s3Encryption.getCustomerKey(), GWConfig.getInstance().getPerformanceMode()); @@ -1354,8 +1628,8 @@ else if (GWConfig.getInstance().isNoDisk()) { objMeta.getObjId(), versionId, length, - GWConstants.FILE_ATTRIBUTE_REPLICATION_REPLICA, - GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, + Constants.FILE_ATTRIBUTE_REPLICATION_REPLICA, + Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, s3Encryption.getCustomerKey(), GWConfig.getInstance().getPerformanceMode()); } @@ -1449,6 +1723,46 @@ else if (GWConfig.getInstance().isNoDisk()) { public boolean deleteObject() throws GWException { OSDClient client = null; + + // check EC exists + File ecFile = new File(KsanUtils.makeECPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId)); + if (ecFile.exists()) { + logger.debug("ec exist : {}", ecFile.getAbsolutePath()); + List ecList = new ArrayList(); + for (DiskPool pool : DiskManager.getInstance().getDiskPoolList()) { + for (Server server : pool.getServerList()) { + for (Disk disk : server.getDiskList()) { + ECPart ecPart = new ECPart(server.getIp(), disk.getPath(), false); + ecList.add(ecPart); + } + } + } + + for (ECPart ecPart : ecList) { + String getPath = KsanUtils.makeECPath(ecPart.getDiskPath(), objMeta.getObjId(), versionId); + if (ecPart.getServerIP().equals(GWUtils.getLocalIP())) { + File file = new File(getPath); + if (file.exists()) { + if (file.delete()) { + logger.debug("delete ec part : {}", getPath); + } else { + logger.debug("fail to delete ec part : {}", getPath); + } + } else { + logger.debug("ec part does not exist.", getPath); + } + } else { + try { + OSDClient ecClient = new OSDClient(ecPart.getServerIP(), (int)GWConfig.getInstance().getOsdPort()); + logger.debug("delete ec part file : {}, to : {}, {}", getPath, ecPart.getServerIP(), ecPart.getDiskPath()); + ecClient.deleteECPart(getPath); + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } + } + } + try { if (objMeta.getReplicaCount() > 1) { if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { @@ -1458,7 +1772,7 @@ public boolean deleteObject() throws GWException { // client = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); client.delete(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId()); // OSDClientManager.getInstance().returnOSDClient(client); - client = null; + // client = null; } if (objMeta.isReplicaExist()) { @@ -1469,7 +1783,7 @@ public boolean deleteObject() throws GWException { // client = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); client.delete(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId()); // OSDClientManager.getInstance().returnOSDClient(client); - client = null; + // client = null; } } @@ -1481,7 +1795,7 @@ public boolean deleteObject() throws GWException { // client = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); client.delete(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId()); // OSDClientManager.getInstance().returnOSDClient(client); - client = null; + // client = null; } } } catch (Exception e) { @@ -1505,102 +1819,23 @@ private void deleteObjectLocal(String path, String objId) throws IOException, GW File file = null; File trashFile = null; if (GWConfig.getInstance().isCacheDiskpath()) { - file = new File(makeCachePath(makeObjPath(path, objId, versionId))); - trashFile = new File(makeCachePath(makeTrashPath(path, objId, versionId))); + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(path, objId, versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(path, objId, versionId))); } else { - file = new File(makeObjPath(path, objId, versionId)); - trashFile = new File(makeTrashPath(path, objId, versionId)); + file = new File(KsanUtils.makeObjPath(path, objId, versionId)); + trashFile = new File(KsanUtils.makeTrashPath(path, objId, versionId)); } updateBucketUsed(objMeta.getBucket(), file.length() * objMeta.getReplicaCount() * -1); if (file.exists()) { retryRenameTo(file, trashFile); if (GWConfig.getInstance().isCacheDiskpath()) { - File link = new File(makeObjPath(path, objId, versionId)); + File link = new File(KsanUtils.makeObjPath(path, objId, versionId)); link.delete(); } } } - // public S3Object uploadPart(String path, long length) throws GWException { - // S3Object s3Object = new S3Object(); - - // if (s3Encryption.isEncryptionEnabled()) { - // CtrCryptoOutputStream encryptOS = null; - - // try { - // MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5); - // byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; - // int readLength = 0; - // long totalReads = 0L; - - // File tmpFile = new File(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); - // com.google.common.io.Files.createParentDirs(tmpFile); - // try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { - // encryptOS = GWUtils.initCtrEncrypt(fos, s3Encryption.getCustomerKey()); - // while ((readLength = s3Parameter.getInputStream().read(buffer, 0, GWConstants.BUFSIZE)) != -1) { - // totalReads += readLength; - // encryptOS.write(buffer, 0, readLength); - // md5er.update(buffer, 0, readLength); - // if (totalReads >= length) { - // break; - // } - // } - // encryptOS.flush(); - // encryptOS.close(); - // } - - // byte[] digest = md5er.digest(); - // String eTag = base16().lowerCase().encode(digest); - - // s3Object.setEtag(eTag); - // s3Object.setLastModified(new Date()); - // s3Object.setFileSize(length); - // s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); - // } catch (Exception e) { - // PrintStack.logging(logger, e); - // throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); - // } - // } else { - // try { - // MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5); - // byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; - // int readLength = 0; - // long totalReads = 0L; - - // File tmpFile = new File(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); - // logger.debug("tmpFile: " + tmpFile.getAbsolutePath()); - // com.google.common.io.Files.createParentDirs(tmpFile); - // try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { - // while ((readLength = s3Parameter.getInputStream().read(buffer, 0, GWConstants.BUFSIZE)) != -1) { - // totalReads += readLength; - // fos.write(buffer, 0, readLength); - // md5er.update(buffer, 0, readLength); - // if (totalReads >= length) { - // break; - // } - // } - // fos.flush(); - // } - - // logger.debug("Total read : {}", totalReads); - - // byte[] digest = md5er.digest(); - // String eTag = base16().lowerCase().encode(digest); - - // s3Object.setEtag(eTag); - // s3Object.setLastModified(new Date()); - // s3Object.setFileSize(length); - // s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); - // } catch (Exception e) { - // PrintStack.logging(logger, e); - // throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); - // } - // } - - // return s3Object; - // } - public S3Object uploadPart(String path, long length) throws GWException { S3Object s3Object = new S3Object(); MessageDigest md5er = null; @@ -1613,162 +1848,354 @@ public S3Object uploadPart(String path, long length) throws GWException { InputStream is = s3Parameter.getInputStream(); if (s3Encryption.isEncryptionEnabled()) { - CtrCryptoOutputStream encryptOS = null; - try { - md5er = MessageDigest.getInstance(GWConstants.MD5); + // s3Object = uploadPartEncription(path, length, is); + s3Object = uploadPartNormal(path, length, is); + } else { + s3Object = uploadPartNormal(path, length, is); + } + + return s3Object; + } + + private S3Object uploadPartNormal(String path, long length, InputStream is) throws GWException { + S3Object s3Object = new S3Object(); + MessageDigest md5er = null; + byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; + int readLength = 0; + long totalReads = 0L; + File filePrimary = null; + File fileReplica = null; + FileOutputStream fosPrimary = null; + FileOutputStream fosReplica = null; + OSDClient clientPrimary = null; + OSDClient clientReplica = null; + + try { + md5er = MessageDigest.getInstance(GWConstants.MD5, new OpenSSL4JProvider()); + if (objMeta.getReplicaCount() > 1) { if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - file = new File(makeCachePath(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); + filePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); } else { - file = new File(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); + filePrimary = new File(KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); } - logger.debug("upload part file : " + file.getAbsolutePath()); - com.google.common.io.Files.createParentDirs(file); - fos = new FileOutputStream(file, false); - encryptOS = GWUtils.initCtrEncrypt(fos, s3Encryption.getCustomerKey()); + com.google.common.io.Files.createParentDirs(filePrimary); + fosPrimary = new FileOutputStream(filePrimary, false); } else { - osdClient = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); - logger.info("osd - {},{}", osdClient.getSocket().getRemoteSocketAddress().toString(), osdClient.getSocket().getLocalPort()); - osdClient.partInit(objMeta.getPrimaryDisk().getPath(), - objMeta.getObjId(), - s3Parameter.getPartNumber(), - length, - s3Encryption.getCustomerKey()); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (clientPrimary == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + clientPrimary = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientPrimary.getSocket().getRemoteSocketAddress().toString(), clientPrimary.getSocket().getLocalPort()); + clientPrimary.partInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + s3Parameter.getPartNumber(), + length, + GWConstants.EMPTY_STRING); } - while ((readLength = is.read(buffer, 0, GWConstants.BUFSIZE)) != -1) { + if (objMeta.isReplicaExist()) { + if (GWUtils.getLocalIP().equals(objMeta.getReplicaDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + fileReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); + } else { + fileReplica = new File(KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); + } + com.google.common.io.Files.createParentDirs(fileReplica); + fosReplica = new FileOutputStream(fileReplica, false); + } else { + // clientReplica = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // if (clientReplica == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getReplicaDisk().getOsdIp()); + // clientReplica = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // } + clientReplica = new OSDClient(objMeta.getReplicaDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientReplica.getSocket().getRemoteSocketAddress().toString(), clientReplica.getSocket().getLocalPort()); + clientReplica.partInit(objMeta.getReplicaDisk().getPath(), + objMeta.getObjId(), + s3Parameter.getPartNumber(), + length, + GWConstants.EMPTY_STRING); + } + } + + while ((readLength = is.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { totalReads += readLength; - if (file == null) { - osdClient.part(buffer, 0, readLength); + if (filePrimary == null) { + clientPrimary.put(buffer, 0, readLength); } else { - encryptOS.write(buffer, 0, readLength); + fosPrimary.write(buffer, 0, readLength); } - + + if (objMeta.isReplicaExist()) { + if (fileReplica == null) { + clientReplica.put(buffer, 0, readLength); + } else { + fosReplica.write(buffer, 0, readLength); + } + } + md5er.update(buffer, 0, readLength); - if (totalReads >= length) { break; } } - if (encryptOS != null) { - encryptOS.flush(); - encryptOS.close(); + + if (filePrimary == null) { + clientPrimary.putFlush(); + clientPrimary = null; + } else { + fosPrimary.flush(); + fosPrimary.close(); } - // try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { - // encryptOS = GWUtils.initCtrEncrypt(fos, s3Encryption.getCustomerKey()); - // while ((readLength = s3Parameter.getInputStream().read(buffer, 0, GWConstants.BUFSIZE)) != -1) { - // totalReads += readLength; - // encryptOS.write(buffer, 0, readLength); - // md5er.update(buffer, 0, readLength); - // if (totalReads >= length) { - // break; - // } - // } - // encryptOS.flush(); - // encryptOS.close(); - // } - logger.debug("Total read : {}", totalReads); - - byte[] digest = md5er.digest(); - String eTag = base16().lowerCase().encode(digest); - - s3Object.setEtag(eTag); - s3Object.setLastModified(new Date()); - s3Object.setFileSize(length); - s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); - } catch (Exception e) { - PrintStack.logging(logger, e); - throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + if (objMeta.isReplicaExist()) { + if (fileReplica == null) { + clientReplica.putFlush(); + clientReplica = null; + } else { + fosReplica.flush(); + fosReplica.close(); + } + } + } else { + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + filePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); + } else { + filePrimary = new File(KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); + } + com.google.common.io.Files.createParentDirs(filePrimary); + fosPrimary = new FileOutputStream(filePrimary, false); + } else { + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (clientPrimary == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + clientPrimary = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientPrimary.getSocket().getRemoteSocketAddress().toString(), clientPrimary.getSocket().getLocalPort()); + clientPrimary.partInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + s3Parameter.getPartNumber(), + length, + GWConstants.EMPTY_STRING); + } + + while ((readLength = is.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + totalReads += readLength; + if (filePrimary == null) { + clientPrimary.put(buffer, 0, readLength); + } else { + fosPrimary.write(buffer, 0, readLength); + } + + md5er.update(buffer, 0, readLength); + if (totalReads >= length) { + break; + } + } + + if (filePrimary == null) { + clientPrimary.putFlush(); + clientPrimary = null; + } else { + fosPrimary.flush(); + fosPrimary.close(); + } } - } else { - try { - md5er = MessageDigest.getInstance(GWConstants.MD5); + + logger.debug("Total read : {}", totalReads); + + byte[] digest = md5er.digest(); + String eTag = base16().lowerCase().encode(digest); + + s3Object.setEtag(eTag); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(length); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + return s3Object; + } + + private S3Object uploadPartEncription(String path, long length, InputStream is) throws GWException { + S3Object s3Object = new S3Object(); + MessageDigest md5er = null; + byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; + int readLength = 0; + long totalReads = 0L; + File filePrimary = null; + File fileReplica = null; + FileOutputStream fosPrimary = null; + FileOutputStream fosReplica = null; + OSDClient clientPrimary = null; + OSDClient clientReplica = null; + CtrCryptoOutputStream encryptPrimary = null; + CtrCryptoOutputStream encryptReplica = null; + + try { + md5er = MessageDigest.getInstance(GWConstants.MD5, new OpenSSL4JProvider()); + if (objMeta.getReplicaCount() > 1) { if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { if (GWConfig.getInstance().isCacheDiskpath()) { - file = new File(makeCachePath(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); + filePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); } else { - file = new File(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); + filePrimary = new File(KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); } - logger.debug("upload part tmpFile : " + file.getAbsolutePath()); - com.google.common.io.Files.createParentDirs(file); - fos = new FileOutputStream(file, false); + com.google.common.io.Files.createParentDirs(filePrimary); + fosPrimary = new FileOutputStream(filePrimary, false); + encryptPrimary = GWUtils.initCtrEncrypt(fosPrimary, s3Encryption.getCustomerKey()); } else { - osdClient = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); - logger.info("osd - {},{}", osdClient.getSocket().getRemoteSocketAddress().toString(), osdClient.getSocket().getLocalPort()); - osdClient.partInit(objMeta.getPrimaryDisk().getPath(), - objMeta.getObjId(), - s3Parameter.getPartNumber(), - length, - GWConstants.EMPTY_STRING); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (clientPrimary == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + clientPrimary = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientPrimary.getSocket().getRemoteSocketAddress().toString(), clientPrimary.getSocket().getLocalPort()); + clientPrimary.partInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + s3Parameter.getPartNumber(), + length, + s3Encryption.getCustomerKey()); } - while ((readLength = is.read(buffer, 0, GWConstants.BUFSIZE)) != -1) { + if (objMeta.isReplicaExist()) { + if (GWUtils.getLocalIP().equals(objMeta.getReplicaDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + fileReplica = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); + } else { + fileReplica = new File(KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); + } + com.google.common.io.Files.createParentDirs(fileReplica); + fosReplica = new FileOutputStream(fileReplica, false); + encryptReplica = GWUtils.initCtrEncrypt(fosReplica, s3Encryption.getCustomerKey()); + } else { + // clientReplica = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // if (clientReplica == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getReplicaDisk().getOsdIp()); + // clientReplica = OSDClientManager.getInstance().getOSDClient(objMeta.getReplicaDisk().getOsdIp()); + // } + clientReplica = new OSDClient(objMeta.getReplicaDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientReplica.getSocket().getRemoteSocketAddress().toString(), clientReplica.getSocket().getLocalPort()); + clientReplica.partInit(objMeta.getReplicaDisk().getPath(), + objMeta.getObjId(), + s3Parameter.getPartNumber(), + length, + s3Encryption.getCustomerKey()); + } + } + + while ((readLength = is.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { totalReads += readLength; - if (file == null) { - osdClient.part(buffer, 0, readLength); + if (filePrimary == null) { + clientPrimary.put(buffer, 0, readLength); } else { - fos.write(buffer, 0, readLength); + encryptPrimary.write(buffer, 0, readLength); } - + + if (objMeta.isReplicaExist()) { + if (fileReplica == null) { + clientReplica.put(buffer, 0, readLength); + } else { + encryptReplica.write(buffer, 0, readLength); + } + } + md5er.update(buffer, 0, readLength); - if (totalReads >= length) { break; } } - if (file != null) { - fos.flush(); - fos.close(); + + if (filePrimary == null) { + clientPrimary.putFlush(); + clientPrimary = null; + } else { + encryptPrimary.flush(); + encryptPrimary.close(); } - // try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { - // while ((readLength = s3Parameter.getInputStream().read(buffer, 0, GWConstants.BUFSIZE)) != -1) { - // totalReads += readLength; - // fos.write(buffer, 0, readLength); - // md5er.update(buffer, 0, readLength); - // if (totalReads >= length) { - // break; - // } - // } - // fos.flush(); - // } - logger.debug("Total read : {}", totalReads); - - byte[] digest = md5er.digest(); - String eTag = base16().lowerCase().encode(digest); + if (objMeta.isReplicaExist()) { + if (fileReplica == null) { + clientReplica.putFlush(); + clientReplica = null; + } else { + encryptReplica.flush(); + encryptReplica.close(); + } + } + } else { + if (GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + if (GWConfig.getInstance().isCacheDiskpath()) { + filePrimary = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); + } else { + filePrimary = new File(KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); + } + com.google.common.io.Files.createParentDirs(filePrimary); + fosPrimary = new FileOutputStream(filePrimary, false); + encryptPrimary = GWUtils.initCtrEncrypt(fosPrimary, s3Encryption.getCustomerKey()); + } else { + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // if (clientPrimary == null) { + // OSDClientManager.getInstance().addClient((int)GWConfig.getInstance().getOsdPort(), (int)GWConfig.getInstance().getOsdClientCount(), objMeta.getPrimaryDisk().getOsdIp()); + // clientPrimary = OSDClientManager.getInstance().getOSDClient(objMeta.getPrimaryDisk().getOsdIp()); + // } + clientPrimary = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + logger.info("osd - {},{}", clientPrimary.getSocket().getRemoteSocketAddress().toString(), clientPrimary.getSocket().getLocalPort()); + clientPrimary.partInit(objMeta.getPrimaryDisk().getPath(), + objMeta.getObjId(), + s3Parameter.getPartNumber(), + length, + s3Encryption.getCustomerKey()); + } + + while ((readLength = is.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + totalReads += readLength; + if (filePrimary == null) { + clientPrimary.put(buffer, 0, readLength); + } else { + encryptPrimary.write(buffer, 0, readLength); + } - s3Object.setEtag(eTag); - s3Object.setLastModified(new Date()); - s3Object.setFileSize(length); - s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); - } catch (Exception e) { - PrintStack.logging(logger, e); - throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + md5er.update(buffer, 0, readLength); + if (totalReads >= length) { + break; + } + } + + if (filePrimary == null) { + clientPrimary.putFlush(); + clientPrimary = null; + } else { + encryptPrimary.flush(); + encryptPrimary.close(); + } } + + logger.debug("Total read : {}", totalReads); + + byte[] digest = md5er.digest(); + String eTag = base16().lowerCase().encode(digest); + + s3Object.setEtag(eTag); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(length); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); + } catch (Exception e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } - + return s3Object; } - // public void deletePart(String diskID) throws Exception { - // String host = DiskManager.getInstance().getOSDIP(diskID); - // if (host == null) { - // logger.error(GWConstants.LOG_S3OBJECT_OPERATION_DISK_IP_NULL, diskID); - // return; - // } - - // String path = DiskManager.getInstance().getLocalPath(diskID); - // if (path == null) { - // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); - // client.deletePart(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), s3Parameter.getPartNumber()); - // OSDClientManager.getInstance().returnOSDClient(client); - // } else { - // File tmpFile = new File(makeTempPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); - // tmpFile.delete(); - // } - // } - public void deletePart(String diskID) throws Exception { String host = DiskManager.getInstance().getOSDIP(diskID); if (host == null) { @@ -1785,9 +2212,9 @@ public void deletePart(String diskID) throws Exception { } else { File tmpFile = null; if (GWConfig.getInstance().isCacheDiskpath()) { - tmpFile = new File(makeCachePath(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); } else { - tmpFile = new File(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); + tmpFile = new File(KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); } if (tmpFile.exists()) { @@ -1798,139 +2225,25 @@ public void deletePart(String diskID) throws Exception { } } - // public S3Object completeMultipart(SortedMap listPart) throws Exception { - // S3Object s3Object = new S3Object(); - // byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; - // MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5); - // long totalLength = 0L; - // long existFileSize = 0L; - // long putSize = 0L; - // long calSize = 0L; - // CtrCryptoOutputStream encryptOS = null; - // CtrCryptoInputStream encryptIS = null; - - // String path = DiskManager.getInstance().getLocalPath(); - - // if (path == null) { - // logger.error(GWConstants.LOG_CANNOT_FIND_LOCAL_PATH); - // throw new GWException(GWErrorCode.INTERNAL_SERVER_ERROR, s3Parameter); - // } - // File tmpFile = new File(makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId)); - // com.google.common.io.Files.createParentDirs(tmpFile); - - // try (FileOutputStream tmpOut = new FileOutputStream(tmpFile)) { - // if (s3Encryption.isEncryptionEnabled()) { - // encryptOS = GWUtils.initCtrEncrypt(tmpOut, s3Encryption.getCustomerKey()); - // // for each part object - // for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { - // Map.Entry entry = it.next(); - // String partPath = DiskManager.getInstance().getLocalPath(entry.getValue().getDiskID()); - // if (!Strings.isNullOrEmpty(partPath)) { - // // part is in local disk - // logger.debug("part : {}, diskID : {}, part path : {}", entry.getKey(), entry.getValue().getDiskID(), partPath); - // File partFile = new File(makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); - // try (FileInputStream fis = new FileInputStream(partFile)) { - // encryptIS = GWUtils.initCtrDecrypt(fis, s3Encryption.getCustomerKey()); - // int readLength = 0; - // while ((readLength = encryptIS.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { - // totalLength += readLength; - // encryptOS.write(buffer, 0, readLength); - // md5er.update(buffer, 0, readLength); - // } - // encryptOS.flush(); - // } - // } else { - // partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); - // String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); - // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); - // client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), encryptOS, md5er); - // totalLength += client.getPart(); - // OSDClientManager.getInstance().returnOSDClient(client); - // } - // } - // } else { - // // for each part object - // for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { - // Map.Entry entry = it.next(); - // String partPath = DiskManager.getInstance().getLocalPath(entry.getValue().getDiskID()); - // if (!Strings.isNullOrEmpty(partPath)) { - // // part is in local disk - // logger.debug("part : {}, diskID : {}, part path : {}", entry.getKey(), entry.getValue().getDiskID(), partPath); - // File partFile = new File(makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); - // try (FileInputStream fis = new FileInputStream(partFile)) { - // int readLength = 0; - // while ((readLength = fis.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { - // totalLength += readLength; - // tmpOut.write(buffer, 0, readLength); - // md5er.update(buffer, 0, readLength); - // } - // tmpOut.flush(); - // } - // } else { - // partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); - // String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); - // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); - // client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), tmpOut, md5er); - // totalLength += client.getPart(); - // OSDClientManager.getInstance().returnOSDClient(client); - // } - // } - // } - // byte[] digest = md5er.digest(); - // String eTag = base16().lowerCase().encode(digest); - - // s3Object.setEtag(eTag); - // s3Object.setLastModified(new Date()); - // s3Object.setFileSize(totalLength); - // s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); - // } - - // OSDClient clientPrimary = null; - // OSDClient clientReplica = null; - // int readLength = 0; - - // existFileSize = objMeta.getSize(); - // putSize = totalLength; - - // tmpFile = new File(makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId)); - // s3Meta.setContentLength(tmpFile.length()); - // logger.info("tmpfile path : {}", tmpFile.getAbsolutePath()); - - // try (FileInputStream fis = new FileInputStream(tmpFile);) { - // if (s3Encryption.isEncryptionEnabled()) { - // s3Object = putObjectEncryption(s3Meta.getContentLength(), fis); - // } else { - // s3Object = putObjectNormal(s3Meta.getContentLength(), fis); - // } - // } catch (Exception e) { - // PrintStack.logging(logger, e); - // } - - // // delete part file - // abortMultipart(listPart); - // tmpFile.delete(); - - // calSize = putSize - existFileSize; - // updateBucketUsed(objMeta.getBucket(), calSize); - - // return s3Object; - // } - public S3Object completeMultipart(SortedMap listPart) throws Exception { S3Object s3Object = new S3Object(); byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; - MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5); + MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5, new OpenSSL4JProvider()); long totalLength = 0L; long existFileSize = 0L; long putSize = 0L; long calSize = 0L; CtrCryptoOutputStream encryptOS = null; CtrCryptoInputStream encryptIS = null; + File file = null; + File tmpFile = null; + File trashFile = null; + boolean isCacheDiskpath = false; String path = objMeta.getPrimaryDisk().getPath(); if (!GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { - logger.info(GWConstants.LOG_CANNOT_FIND_LOCAL_PATH); + logger.info(GWConstants.LOG_CANNOT_FIND_LOCAL_PATH, objMeta.getPrimaryDisk().getId()); String partInfos = GWConstants.EMPTY_STRING; for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { Map.Entry entry = it.next(); @@ -1940,131 +2253,301 @@ public S3Object completeMultipart(SortedMap listPart) throws Exce } logger.debug("partInfos : {}", partInfos); OSDClient client = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); - OsdData data = client.completeMultipart(path, objMeta.getObjId(), versionId, s3Encryption.getCustomerKey(), partInfos); + OsdData data = null; + if (objMeta.getReplicaCount() > 1) { + data = client.completeMultipart(path, objMeta.getObjId(), versionId, s3Encryption.getCustomerKey(), Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId(), partInfos); + } else { + data = client.completeMultipart(path, objMeta.getObjId(), versionId, s3Encryption.getCustomerKey(), Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, partInfos); + } + s3Object.setEtag(data.getETag()); s3Object.setLastModified(new Date()); s3Object.setFileSize(data.getFileSize()); s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); - - return s3Object; - } - - File tmpFile = null; - if (GWConfig.getInstance().isCacheDiskpath()) { - tmpFile = new File(makeCachePath(makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId))); + // return s3Object; } else { - tmpFile = new File(makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId)); - } - com.google.common.io.Files.createParentDirs(tmpFile); - - try (FileOutputStream tmpOut = new FileOutputStream(tmpFile)) { - if (s3Encryption.isEncryptionEnabled()) { - encryptOS = GWUtils.initCtrEncrypt(tmpOut, s3Encryption.getCustomerKey()); - // for each part object - for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { - Map.Entry entry = it.next(); - String partPath = DiskManager.getInstance().getLocalPath(entry.getValue().getDiskID()); - if (!Strings.isNullOrEmpty(partPath)) { - // part is in local disk - logger.debug("part : {}, diskID : {}, part path : {}", entry.getKey(), entry.getValue().getDiskID(), partPath); - File partFile = null; - if (GWConfig.getInstance().isCacheDiskpath()) { - partFile = new File(makeCachePath(makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber())))); + if (GWConfig.getInstance().isCacheDiskpath()) { + isCacheDiskpath = true; + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(path, objMeta.getObjId(), versionId))); + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(path, objMeta.getObjId(), versionId))); + } else { + file = new File(KsanUtils.makeObjPath(path, objMeta.getObjId(), versionId)); + tmpFile = new File(KsanUtils.makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId)); + trashFile = new File(KsanUtils.makeTrashPath(path, objMeta.getObjId(), versionId)); + } + com.google.common.io.Files.createParentDirs(file); + com.google.common.io.Files.createParentDirs(tmpFile); + + try (FileOutputStream tmpOut = new FileOutputStream(tmpFile)) { + if (s3Encryption.isEncryptionEnabled()) { + encryptOS = GWUtils.initCtrEncrypt(tmpOut, s3Encryption.getCustomerKey()); + // for each part object + for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { + Map.Entry entry = it.next(); + String partPath = DiskManager.getInstance().getLocalPath(entry.getValue().getDiskID()); + if (!Strings.isNullOrEmpty(partPath)) { + // part is in local disk + logger.debug("part : {}, diskID : {}, part path : {}", entry.getKey(), entry.getValue().getDiskID(), partPath); + File partFile = null; + if (GWConfig.getInstance().isCacheDiskpath()) { + partFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber())))); + } else { + partFile = new File(KsanUtils.makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); + } + + totalLength += partFile.length(); + logger.debug("totalLength: {}", totalLength); + try (FileInputStream fis = new FileInputStream(partFile)) { + // encryptIS = GWUtils.initCtrDecrypt(fis, s3Encryption.getCustomerKey()); + int readLength = 0; + while ((readLength = fis.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + // totalLength += readLength; + encryptOS.write(buffer, 0, readLength); + // md5er.update(buffer, 0, readLength); + } + encryptOS.flush(); + // encryptIS.close(); + } } else { - partFile = new File(makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); + partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); + String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); + OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); + client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), encryptOS, md5er); + totalLength += client.getPart(); + + // partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); + // String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); + // OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); + // // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); + // client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), encryptOS, md5er); + // totalLength += client.getPart(); + // OSDClientManager.getInstance().returnOSDClient(client); } - - try (FileInputStream fis = new FileInputStream(partFile)) { - encryptIS = GWUtils.initCtrDecrypt(fis, s3Encryption.getCustomerKey()); - int readLength = 0; - while ((readLength = encryptIS.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { - totalLength += readLength; - encryptOS.write(buffer, 0, readLength); - md5er.update(buffer, 0, readLength); + } + encryptOS.close(); + } else { + // for each part object + for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { + Map.Entry entry = it.next(); + String partPath = DiskManager.getInstance().getLocalPath(entry.getValue().getDiskID()); + if (!Strings.isNullOrEmpty(partPath)) { + // part is in local disk + logger.debug("part : {}, diskID : {}, part path : {}", entry.getKey(), entry.getValue().getDiskID(), partPath); + File partFile = null; + if (GWConfig.getInstance().isCacheDiskpath()) { + partFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber())))); + } else { + partFile = new File(KsanUtils.makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); + } + totalLength += partFile.length(); + logger.debug("totalLength: {}", totalLength); + try (FileInputStream fis = new FileInputStream(partFile)) { + int readLength = 0; + while ((readLength = fis.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + // totalLength += readLength; + tmpOut.write(buffer, 0, readLength); + // md5er.update(buffer, 0, readLength); + } + tmpOut.flush(); } - encryptOS.flush(); + } else { + partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); + String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); + OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); + client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), tmpOut, md5er); + totalLength += client.getPart(); + + // partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); + // String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); + // OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); + // // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); + // client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), tmpOut, md5er); + // totalLength += client.getPart(); + // OSDClientManager.getInstance().returnOSDClient(client); } - } else { - partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); - String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); - OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); - // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); - client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), encryptOS, md5er); - totalLength += client.getPart(); - // OSDClientManager.getInstance().returnOSDClient(client); } + tmpOut.close(); } - } else { - // for each part object + + if (file.exists()) { + File tempFile = new File(file.getAbsolutePath()); + retryRenameTo(tempFile, trashFile); + } + // if (objMeta.getReplicaDisk() != null && !Strings.isNullOrEmpty(objMeta.getReplicaDisk().getId())) { + // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId()); + // } else { + // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + // } + if (objMeta.getReplicaCount() > 1) { + KsanUtils.setAttributeFileReplication(tmpFile, Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId()); + } else { + KsanUtils.setAttributeFileReplication(tmpFile, Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + } + retryRenameTo(tmpFile, file); + if (isCacheDiskpath) { + String filePath = KsanUtils.makeObjPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), versionId); + Files.createSymbolicLink(Paths.get(filePath), Paths.get(file.getAbsolutePath())); + } + + // s3Object.setEtag(eTag); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(totalLength); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); + } + } + + if (objMeta.getReplicaCount() > 1) { + if (!GWUtils.getLocalIP().equals(objMeta.getReplicaDisk().getOsdIp())) { + logger.info(GWConstants.LOG_CANNOT_FIND_LOCAL_PATH, objMeta.getPrimaryDisk().getId()); + String partInfos = GWConstants.EMPTY_STRING; for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { Map.Entry entry = it.next(); - String partPath = DiskManager.getInstance().getLocalPath(entry.getValue().getDiskID()); - if (!Strings.isNullOrEmpty(partPath)) { - // part is in local disk - logger.debug("part : {}, diskID : {}, part path : {}", entry.getKey(), entry.getValue().getDiskID(), partPath); - File partFile = null; - if (GWConfig.getInstance().isCacheDiskpath()) { - partFile = new File(makeCachePath(makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber())))); - } else { - partFile = new File(makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); - } - try (FileInputStream fis = new FileInputStream(partFile)) { - int readLength = 0; - while ((readLength = fis.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { - totalLength += readLength; - tmpOut.write(buffer, 0, readLength); - md5er.update(buffer, 0, readLength); + partInfos += entry.getValue().getPartNumber() + GWConstants.SLASH + + entry.getValue().getDiskID() + GWConstants.SLASH + + entry.getValue().getPartSize() + GWConstants.COMMA; + } + logger.debug("partInfos : {}", partInfos); + OSDClient client = new OSDClient(objMeta.getReplicaDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + // OsdData data = client.completeMultipart(path, objMeta.getObjId(), versionId, s3Encryption.getCustomerKey(), partInfos); + OsdData data = client.completeMultipart(path, objMeta.getObjId(), versionId, s3Encryption.getCustomerKey(), Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL, partInfos); + s3Object.setEtag(data.getETag()); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(data.getFileSize()); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); + // return s3Object; + } else { + if (GWConfig.getInstance().isCacheDiskpath()) { + isCacheDiskpath = true; + file = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeObjPath(path, objMeta.getObjId(), versionId))); + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId))); + trashFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTrashPath(path, objMeta.getObjId(), versionId))); + } else { + file = new File(KsanUtils.makeObjPath(path, objMeta.getObjId(), versionId)); + tmpFile = new File(KsanUtils.makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId)); + trashFile = new File(KsanUtils.makeTrashPath(path, objMeta.getObjId(), versionId)); + } + com.google.common.io.Files.createParentDirs(file); + com.google.common.io.Files.createParentDirs(tmpFile); + + try (FileOutputStream tmpOut = new FileOutputStream(tmpFile)) { + if (s3Encryption.isEncryptionEnabled()) { + encryptOS = GWUtils.initCtrEncrypt(tmpOut, s3Encryption.getCustomerKey()); + // for each part object + for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { + Map.Entry entry = it.next(); + String partPath = DiskManager.getInstance().getLocalPath(entry.getValue().getDiskID()); + if (!Strings.isNullOrEmpty(partPath)) { + // part is in local disk + logger.debug("part : {}, diskID : {}, part path : {}", entry.getKey(), entry.getValue().getDiskID(), partPath); + File partFile = null; + if (GWConfig.getInstance().isCacheDiskpath()) { + partFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber())))); + } else { + partFile = new File(KsanUtils.makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); + } + + totalLength += partFile.length(); + logger.debug("totalLength: {}", totalLength); + try (FileInputStream fis = new FileInputStream(partFile)) { + // encryptIS = GWUtils.initCtrDecrypt(fis, s3Encryption.getCustomerKey()); + int readLength = 0; + while ((readLength = fis.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + // totalLength += readLength; + encryptOS.write(buffer, 0, readLength); + // md5er.update(buffer, 0, readLength); + } + encryptOS.flush(); + // encryptIS.close(); + } + } else { + partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); + String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); + OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); + client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), encryptOS, md5er); + totalLength += client.getPart(); + + // partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); + // String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); + // OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); + // // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); + // client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), encryptOS, md5er); + // totalLength += client.getPart(); + // OSDClientManager.getInstance().returnOSDClient(client); } - tmpOut.flush(); } + encryptOS.close(); } else { - partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); - String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); - OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); - // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); - client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), tmpOut, md5er); - totalLength += client.getPart(); - // OSDClientManager.getInstance().returnOSDClient(client); + // for each part object + for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { + Map.Entry entry = it.next(); + String partPath = DiskManager.getInstance().getLocalPath(entry.getValue().getDiskID()); + if (!Strings.isNullOrEmpty(partPath)) { + // part is in local disk + logger.debug("part : {}, diskID : {}, part path : {}", entry.getKey(), entry.getValue().getDiskID(), partPath); + File partFile = null; + if (GWConfig.getInstance().isCacheDiskpath()) { + partFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber())))); + } else { + partFile = new File(KsanUtils.makeTempPartPath(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); + } + totalLength += partFile.length(); + logger.debug("totalLength: {}", totalLength); + try (FileInputStream fis = new FileInputStream(partFile)) { + int readLength = 0; + while ((readLength = fis.read(buffer, 0, GWConstants.MAXBUFSIZE)) != -1) { + // totalLength += readLength; + tmpOut.write(buffer, 0, readLength); + // md5er.update(buffer, 0, readLength); + } + tmpOut.flush(); + } + } else { + partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); + String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); + OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); + client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), tmpOut, md5er); + totalLength += client.getPart(); + + // partPath = DiskManager.getInstance().getPath(entry.getValue().getDiskID()); + // String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); + // OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); + // // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); + // client.getPartInit(partPath, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()), entry.getValue().getPartSize(), tmpOut, md5er); + // totalLength += client.getPart(); + // OSDClientManager.getInstance().returnOSDClient(client); + } + } + tmpOut.close(); + } + + if (file.exists()) { + File tempFile = new File(file.getAbsolutePath()); + retryRenameTo(tempFile, trashFile); + } + // if (objMeta.getReplicaDisk() != null && !Strings.isNullOrEmpty(objMeta.getReplicaDisk().getId())) { + // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, objMeta.getReplicaDisk().getId()); + // } else { + // setAttributeFileReplication(tmpFilePrimary, GWConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY, GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + // } + KsanUtils.setAttributeFileReplication(tmpFile, Constants.FILE_ATTRIBUTE_REPLICATION_REPLICA, Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL); + retryRenameTo(tmpFile, file); + if (isCacheDiskpath) { + String filePath = KsanUtils.makeObjPath(objMeta.getReplicaDisk().getPath(), objMeta.getObjId(), versionId); + Files.createSymbolicLink(Paths.get(filePath), Paths.get(file.getAbsolutePath())); } + + // s3Object.setEtag(eTag); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(totalLength); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); } } - byte[] digest = md5er.digest(); - String eTag = base16().lowerCase().encode(digest); - - s3Object.setEtag(eTag); - s3Object.setLastModified(new Date()); - s3Object.setFileSize(totalLength); - s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); } - - OSDClient clientPrimary = null; - OSDClient clientReplica = null; - int readLength = 0; - existFileSize = objMeta.getSize(); - putSize = totalLength; - - if (GWConfig.getInstance().isCacheDiskpath()) { - tmpFile = new File(makeCachePath(makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId))); - } else { - tmpFile = new File(makeTempCompleteMultipartPath(path, objMeta.getObjId(), versionId)); - } - s3Meta.setContentLength(tmpFile.length()); - logger.info("tmpfile path : {}", tmpFile.getAbsolutePath()); - - try (FileInputStream fis = new FileInputStream(tmpFile);) { - if (s3Encryption.isEncryptionEnabled()) { - s3Object = putObjectEncryption(s3Meta.getContentLength(), fis); - } else { - s3Object = putObjectNormal(s3Meta.getContentLength(), fis); - } - } catch (Exception e) { - PrintStack.logging(logger, e); - } - // delete part file - abortMultipart(listPart); - tmpFile.delete(); + // abortMultipart(listPart); + // tmpFile.delete(); calSize = putSize - existFileSize; updateBucketUsed(objMeta.getBucket(), calSize); @@ -2072,31 +2555,6 @@ public S3Object completeMultipart(SortedMap listPart) throws Exce return s3Object; } - // public void abortMultipart(SortedMap listPart) throws GWException { - // try { - // String path = null; - // for (Iterator> it = listPart.entrySet().iterator(); it.hasNext();) { - // Map.Entry entry = it.next(); - // logger.info("key : {}, diskId : {}", entry.getKey(), objMeta.getPrimaryDisk().getId()); - // path = DiskManager.getInstance().getLocalPath(entry.getValue().getDiskID()); - - // if (!Strings.isNullOrEmpty(path)) { - // // part is in local disk - // File partFile = new File(makeTempPartPath(path, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); - // partFile.delete(); - // } else { - // String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); - // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); - // client.deletePart(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), s3Parameter.getPartNumber()); - // OSDClientManager.getInstance().returnOSDClient(client); - // } - // } - // } catch (Exception e) { - // PrintStack.logging(logger, e); - // throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); - // } - // } - public void abortMultipart(SortedMap listPart) throws GWException { try { String path = null; @@ -2109,16 +2567,16 @@ public void abortMultipart(SortedMap listPart) throws GWException // part is in local disk File partFile = null; if (GWConfig.getInstance().isCacheDiskpath()) { - partFile = new File(makeCachePath(makeTempPartPath(path, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber())))); + partFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(path, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber())))); } else { - partFile = new File(makeTempPartPath(path, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); + partFile = new File(KsanUtils.makeTempPartPath(path, objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber()))); } partFile.delete(); } else { String host = DiskManager.getInstance().getOSDIP(entry.getValue().getDiskID()); OSDClient client = new OSDClient(host, (int)GWConfig.getInstance().getOsdPort()); // OSDClient client = OSDClientManager.getInstance().getOSDClient(host); - client.deletePart(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), s3Parameter.getPartNumber()); + client.deletePart(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), String.valueOf(entry.getValue().getPartNumber())); // OSDClientManager.getInstance().returnOSDClient(client); } } @@ -2128,104 +2586,9 @@ public void abortMultipart(SortedMap listPart) throws GWException } } - // public S3Object uploadPartCopy(String path, Metadata srcObjMeta, S3Range s3Range, S3Encryption srcEncryption) throws ResourceNotFoundException, Exception { - // S3Object s3Object = new S3Object(); - // boolean isTgtEncript = s3Encryption.isEncryptionEnabled(); - // OSDClient client = null; - // String srcKey = srcEncryption.getCustomerKey(); - // long actualSize = 0L; - // CtrCryptoOutputStream encryptOS = null; - // OutputStream outputStream = null; - - // String copySourceRange = GWConstants.EMPTY_STRING; - // if (s3Range != null && s3Range.getListRange().size() > 0) { - // for (S3Range.Range range : s3Range.getListRange()) { - // if (Strings.isNullOrEmpty(copySourceRange)) { - // copySourceRange = String.valueOf(range.getOffset()) + GWConstants.COMMA + String.valueOf(range.getLength()); - // } else { - // copySourceRange += GWConstants.SLASH + String.valueOf(range.getOffset()) + GWConstants.COMMA + String.valueOf(range.getLength()); - // } - // } - // } - - // logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_COPY_SOURCE_RANGE, copySourceRange); - // File tmpFile = new File(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); - // com.google.common.io.Files.createParentDirs(tmpFile); - - // try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { - // if (srcObjMeta.getReplicaCount() > 1) { - // if (GWUtils.getLocalIP().equals(srcObjMeta.getPrimaryDisk().getOsdIp())) { - // actualSize = getObjectLocal(fos, srcObjMeta.getPrimaryDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), copySourceRange, srcKey); - // } else if (GWUtils.getLocalIP().equals(srcObjMeta.getReplicaDisk().getOsdIp())) { - // actualSize = getObjectLocal(fos, srcObjMeta.getReplicaDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), copySourceRange, srcKey); - // } else { - // try { - // client = OSDClientManager.getInstance().getOSDClient(srcObjMeta.getPrimaryDisk().getOsdIp()); - // client.getInit(srcObjMeta.getPrimaryDisk().getPath(), - // srcObjMeta.getObjId(), - // srcObjMeta.getVersionId(), - // srcObjMeta.getSize(), - // copySourceRange, - // fos, - // srcKey); - // actualSize = client.get(); - // OSDClientManager.getInstance().returnOSDClient(client); - // } catch (Exception e) { - // client = OSDClientManager.getInstance().getOSDClient(srcObjMeta.getReplicaDisk().getOsdIp()); - // client.getInit(srcObjMeta.getReplicaDisk().getPath(), - // srcObjMeta.getObjId(), - // srcObjMeta.getVersionId(), - // srcObjMeta.getSize(), - // copySourceRange, - // fos, - // srcKey); - // actualSize = client.get(); - // OSDClientManager.getInstance().returnOSDClient(client); - // } - // } - // } else { - // if (GWUtils.getLocalIP().equals(srcObjMeta.getPrimaryDisk().getOsdIp())) { - // actualSize = getObjectLocal(fos, srcObjMeta.getPrimaryDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), copySourceRange, srcKey); - // } else { - // client = OSDClientManager.getInstance().getOSDClient(srcObjMeta.getPrimaryDisk().getOsdIp()); - // client.getInit(srcObjMeta.getPrimaryDisk().getPath(), - // srcObjMeta.getObjId(), - // srcObjMeta.getVersionId(), - // srcObjMeta.getSize(), - // copySourceRange, - // fos, - // srcKey); - // actualSize = client.get(); - // OSDClientManager.getInstance().returnOSDClient(client); - // } - // } - // } - - // long totalLength = 0L; - // try (FileInputStream is = new FileInputStream(tmpFile)) { - // MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5); - // byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; - // int readLength = 0; - // while ((readLength = is.read(buffer, 0, GWConstants.BUFSIZE)) != -1) { - // totalLength += readLength; - // md5er.update(buffer, 0, readLength); - // } - // byte[] digest = md5er.digest(); - // String eTag = base16().lowerCase().encode(digest); - // s3Object.setEtag(eTag); - // } catch (IOException e) { - // PrintStack.logging(logger, e); - // throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); - // } - - // s3Object.setLastModified(new Date()); - // s3Object.setFileSize(totalLength); - // s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); - // return s3Object; - // } - public S3Object uploadPartCopy(String path, Metadata srcObjMeta, S3Range s3Range, S3Encryption srcEncryption) throws ResourceNotFoundException, Exception { S3Object s3Object = new S3Object(); + MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5, new OpenSSL4JProvider()); boolean isTgtEncript = s3Encryption.isEncryptionEnabled(); OSDClient client = null; String srcKey = srcEncryption.getCustomerKey(); @@ -2245,86 +2608,105 @@ public S3Object uploadPartCopy(String path, Metadata srcObjMeta, S3Range s3Range } logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_COPY_SOURCE_RANGE, copySourceRange); - File tmpFile = null; - if (GWConfig.getInstance().isCacheDiskpath()) { - tmpFile = new File(makeCachePath(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); - } else { - tmpFile = new File(makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); - } - com.google.common.io.Files.createParentDirs(tmpFile); - try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { - if (srcObjMeta.getReplicaCount() > 1) { - if (GWUtils.getLocalIP().equals(srcObjMeta.getPrimaryDisk().getOsdIp())) { - actualSize = getObjectLocal(fos, srcObjMeta.getPrimaryDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), copySourceRange, srcKey); - } else if (GWUtils.getLocalIP().equals(srcObjMeta.getReplicaDisk().getOsdIp())) { - actualSize = getObjectLocal(fos, srcObjMeta.getReplicaDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), copySourceRange, srcKey); + if (!GWUtils.getLocalIP().equals(objMeta.getPrimaryDisk().getOsdIp())) { + logger.info(GWConstants.LOG_CANNOT_FIND_LOCAL_PATH, objMeta.getPrimaryDisk().getId()); + client = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + OsdData data = client.partCopy(srcObjMeta.getPrimaryDisk().getId(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), String.valueOf(srcObjMeta.getSize()), copySourceRange, path, objMeta.getObjId(), s3Parameter.getPartNumber()); + + s3Object.setEtag(data.getETag()); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(data.getFileSize()); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); + // return s3Object; + } else { + File tmpFile = null; + if (GWConfig.getInstance().isCacheDiskpath()) { + tmpFile = new File(GWConfig.getInstance().getCacheDiskpath() + (KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber()))); + } else { + tmpFile = new File(KsanUtils.makeTempPartPath(path, objMeta.getObjId(), s3Parameter.getPartNumber())); + } + com.google.common.io.Files.createParentDirs(tmpFile); + // tmp src object - get src object data + // File tmpFile = new File(KsanUtils.makeTempCopyPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId())); + // com.google.common.io.Files.createParentDirs(tmpFile); + + try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { + if (srcObjMeta.getReplicaCount() > 1) { + if (GWUtils.getLocalIP().equals(srcObjMeta.getPrimaryDisk().getOsdIp())) { + actualSize = getObjectLocal(fos, srcObjMeta.getPrimaryDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), copySourceRange, md5er, srcKey); + } else if (GWUtils.getLocalIP().equals(srcObjMeta.getReplicaDisk().getOsdIp())) { + actualSize = getObjectLocal(fos, srcObjMeta.getReplicaDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), copySourceRange, md5er, srcKey); + } else { + try { + client = new OSDClient(srcObjMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + // client = OSDClientManager.getInstance().getOSDClient(srcObjMeta.getPrimaryDisk().getOsdIp()); + client.getInitWithMD5(srcObjMeta.getPrimaryDisk().getPath(), + srcObjMeta.getObjId(), + srcObjMeta.getVersionId(), + srcObjMeta.getSize(), + copySourceRange, + fos, + md5er, + srcKey); + actualSize = client.getWithMD5(); + // OSDClientManager.getInstance().returnOSDClient(client); + } catch (Exception e) { + client = new OSDClient(srcObjMeta.getReplicaDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + // client = OSDClientManager.getInstance().getOSDClient(srcObjMeta.getReplicaDisk().getOsdIp()); + client.getInitWithMD5(srcObjMeta.getReplicaDisk().getPath(), + srcObjMeta.getObjId(), + srcObjMeta.getVersionId(), + srcObjMeta.getSize(), + copySourceRange, + fos, + md5er, + srcKey); + actualSize = client.getWithMD5(); + // OSDClientManager.getInstance().returnOSDClient(client); + } + } } else { - try { - client = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); + if (GWUtils.getLocalIP().equals(srcObjMeta.getPrimaryDisk().getOsdIp())) { + actualSize = getObjectLocal(fos, srcObjMeta.getPrimaryDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), copySourceRange, srcKey); + } else { + client = new OSDClient(srcObjMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); // client = OSDClientManager.getInstance().getOSDClient(srcObjMeta.getPrimaryDisk().getOsdIp()); - client.getInit(srcObjMeta.getPrimaryDisk().getPath(), + client.getInitWithMD5(srcObjMeta.getPrimaryDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), srcObjMeta.getSize(), copySourceRange, fos, + md5er, srcKey); - actualSize = client.get(); - // OSDClientManager.getInstance().returnOSDClient(client); - } catch (Exception e) { - client = new OSDClient(objMeta.getReplicaDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); - // client = OSDClientManager.getInstance().getOSDClient(srcObjMeta.getReplicaDisk().getOsdIp()); - client.getInit(srcObjMeta.getReplicaDisk().getPath(), - srcObjMeta.getObjId(), - srcObjMeta.getVersionId(), - srcObjMeta.getSize(), - copySourceRange, - fos, - srcKey); - actualSize = client.get(); + actualSize = client.getWithMD5(); // OSDClientManager.getInstance().returnOSDClient(client); } } - } else { - if (GWUtils.getLocalIP().equals(srcObjMeta.getPrimaryDisk().getOsdIp())) { - actualSize = getObjectLocal(fos, srcObjMeta.getPrimaryDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), copySourceRange, srcKey); - } else { - client = new OSDClient(objMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); - // client = OSDClientManager.getInstance().getOSDClient(srcObjMeta.getPrimaryDisk().getOsdIp()); - client.getInit(srcObjMeta.getPrimaryDisk().getPath(), - srcObjMeta.getObjId(), - srcObjMeta.getVersionId(), - srcObjMeta.getSize(), - copySourceRange, - fos, - srcKey); - actualSize = client.get(); - // OSDClientManager.getInstance().returnOSDClient(client); - } - } - } - - long totalLength = 0L; - try (FileInputStream is = new FileInputStream(tmpFile)) { - MessageDigest md5er = MessageDigest.getInstance(GWConstants.MD5); - byte[] buffer = new byte[GWConstants.MAXBUFSIZE]; - int readLength = 0; - while ((readLength = is.read(buffer, 0, GWConstants.BUFSIZE)) != -1) { - totalLength += readLength; - md5er.update(buffer, 0, readLength); } + + logger.debug("Total read : {}", actualSize); + byte[] digest = md5er.digest(); - String eTag = base16().lowerCase().encode(digest); + String eTag = base16().lowerCase().encode(digest); + s3Object.setEtag(eTag); - } catch (IOException e) { - PrintStack.logging(logger, e); - throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + s3Object.setLastModified(new Date()); + s3Object.setFileSize(actualSize); + s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); } + + // long totalLength = 0L; + // tmpFile = new File(KsanUtils.makeTempCopyPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId())); + // try (FileInputStream is = new FileInputStream(tmpFile)) { + // s3Object = uploadPartNormal(path, tmpFile.length(), is); + // } catch (IOException e) { + // PrintStack.logging(logger, e); + // throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + // } + // tmpFile.delete(); - s3Object.setLastModified(new Date()); - s3Object.setFileSize(totalLength); - s3Object.setDeleteMarker(GWConstants.OBJECT_TYPE_FILE); return s3Object; } @@ -2338,14 +2720,15 @@ public S3Object copyObject(Metadata srcObjMeta, S3Encryption srcEncryption) thro try { logger.info(GWConstants.LOG_S3OBJECT_OPERATION_LOCAL_IP, GWUtils.getLocalIP()); logger.info(GWConstants.LOG_S3OBJECT_OPERATION_OBJ_PRIMARY_IP, objMeta.getPrimaryDisk().getOsdIp()); - if (objMeta.getReplicaDisk() != null) { + + if (srcObjMeta.getReplicaCount() > 1 && objMeta.getReplicaDisk() != null) { logger.info(GWConstants.LOG_S3OBJECT_OPERATION_OBJ_REPLICA_IP, objMeta.getReplicaDisk().getOsdIp()); } // tmp src object - get src object data - File tmpFile = new File(makeTempCopyPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId())); + File tmpFile = new File(KsanUtils.makeTempCopyPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId())); com.google.common.io.Files.createParentDirs(tmpFile); - try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { + try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { if (srcObjMeta.getReplicaCount() > 1) { if (GWUtils.getLocalIP().equals(srcObjMeta.getPrimaryDisk().getOsdIp())) { actualSize = getObjectLocal(fos, srcObjMeta.getPrimaryDisk().getPath(), srcObjMeta.getObjId(), srcObjMeta.getVersionId(), GWConstants.EMPTY_STRING, key); @@ -2386,8 +2769,8 @@ public S3Object copyObject(Metadata srcObjMeta, S3Encryption srcEncryption) thro client = new OSDClient(srcObjMeta.getPrimaryDisk().getOsdIp(), (int)GWConfig.getInstance().getOsdPort()); // client = OSDClientManager.getInstance().getOSDClient(srcObjMeta.getPrimaryDisk().getOsdIp()); client.getInit(srcObjMeta.getPrimaryDisk().getPath(), - srcObjMeta.getObjId(), - srcObjMeta.getVersionId(), + srcObjMeta.getObjId(), + srcObjMeta.getVersionId(), fileSize, GWConstants.EMPTY_STRING, fos, @@ -2405,7 +2788,7 @@ public S3Object copyObject(Metadata srcObjMeta, S3Encryption srcEncryption) thro throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } - tmpFile = new File(makeTempCopyPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId())); + tmpFile = new File(KsanUtils.makeTempCopyPath(objMeta.getPrimaryDisk().getPath(), objMeta.getObjId(), objMeta.getVersionId())); s3Meta.setContentLength(tmpFile.length()); logger.info("tmpfile path : {}", tmpFile.getAbsolutePath()); @@ -2423,24 +2806,8 @@ public S3Object copyObject(Metadata srcObjMeta, S3Encryption srcEncryption) thro PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); } - - return s3Object; - } - - private String makeDirectoryName(String objId) { - byte[] path = new byte[6]; - byte[] byteObjId = objId.getBytes(); - path[0] = GWConstants.CHAR_SLASH; - int index = 1; - - path[index++] = byteObjId[0]; - path[index++] = byteObjId[1]; - path[index++] = GWConstants.CHAR_SLASH; - path[index++] = byteObjId[2]; - path[index] = byteObjId[3]; - - return new String(path); + return s3Object; } private void setObjManager() throws Exception { @@ -2451,6 +2818,11 @@ private void releaseObjManager() throws Exception { ObjManagerHelper.getInstance().returnObjManager(objManager); } + // private String GWConfig.getInstance().getCacheDiskpath() + (String path) { + // String fullPath = GWConfig.getInstance().getCacheDiskpath() + path; + // return fullPath; + // } + private void updateBucketUsed(String bucketName, long size) throws GWException { try { setObjManager(); @@ -2468,82 +2840,6 @@ private void updateBucketUsed(String bucketName, long size) throws GWException { } } - private String makeObjPath(String path, String objId, String versionId) { - if (Strings.isNullOrEmpty(versionId)) { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - } - String fullPath = path + GWConstants.SLASH + GWConstants.OBJ_DIR + makeDirectoryName(objId) + GWConstants.SLASH + objId + GWConstants.UNDERSCORE + versionId; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_OBJ_PATH, fullPath); - return fullPath; - } - - private String makeTempPath(String path, String objId, String versionId) { - if (Strings.isNullOrEmpty(versionId)) { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - } - String uuid = UUID.randomUUID().toString(); - String fullPath = path + GWConstants.SLASH + GWConstants.TEMP_DIR + GWConstants.SLASH + objId + GWConstants.UNDERSCORE + uuid + GWConstants.UNDERSCORE + versionId; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_TEMP_PATH, fullPath); - return fullPath; - } - - private String makeTrashPath(String path, String objId, String versionId) { - if (Strings.isNullOrEmpty(versionId)) { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - } - String uuid = UUID.randomUUID().toString(); - String fullPath = path + GWConstants.SLASH + GWConstants.TRASH_DIR + GWConstants.SLASH + objId + GWConstants.UNDERSCORE + versionId + GWConstants.DASH + uuid; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_TRASH_PATH, fullPath); - return fullPath; - } - - private String makeECPath(String path, String objId, String versionId) { - if (Strings.isNullOrEmpty(versionId)) { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - } - String fullPath = path + GWConstants.SLASH + GWConstants.EC_DIR + makeDirectoryName(objId) + GWConstants.SLASH + GWConstants.POINT + objId + GWConstants.UNDERSCORE + versionId; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_EC_PATH, fullPath); - return fullPath; - } - - private String makeTempPartPath(String path, String objId, String partNumber) { - String fullPath = path + GWConstants.SLASH + GWConstants.TEMP_DIR + GWConstants.SLASH + objId + GWConstants.UNDERSCORE + partNumber; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_TEMP_PATH, fullPath); - return fullPath; - } - - private String makeTempCompleteMultipartPath(String path, String objId, String versionId) { - if (Strings.isNullOrEmpty(versionId)) { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - } - String fullPath = path + GWConstants.SLASH + GWConstants.TEMP_COMPLETE_DIR + GWConstants.SLASH + objId + GWConstants.UNDERSCORE + versionId; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_TEMP_PATH, fullPath); - return fullPath; - } - - private String makeTempCopyPath(String path, String objId, String versionId) { - if (Strings.isNullOrEmpty(versionId)) { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - } - String fullPath = path + GWConstants.SLASH + GWConstants.TEMP_COPY_DIR + GWConstants.SLASH + objId + GWConstants.UNDERSCORE + versionId; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_TEMP_PATH, fullPath); - return fullPath; - } - - private String makeECDecodePath(String path, String objId, String versionId) { - if (Strings.isNullOrEmpty(versionId)) { - versionId = GWConstants.VERSIONING_DISABLE_TAIL; - } - String fullPath = path + GWConstants.SLASH + GWConstants.EC_DIR + makeDirectoryName(objId) + GWConstants.SLASH + objId + GWConstants.UNDERSCORE + versionId; - logger.debug(GWConstants.LOG_S3OBJECT_OPERATION_EC_PATH, fullPath); - return fullPath; - } - - private String makeCachePath(String path) { - String fullPath = GWConfig.getInstance().getCacheDiskpath() + path; - return fullPath; - } - private void retryRenameTo(File srcFile, File destFile) throws IOException { if (srcFile.exists()) { for (int i = 0; i < GWConstants.RETRY_COUNT; i++) { @@ -2560,50 +2856,4 @@ private void retryRenameTo(File srcFile, File destFile) throws IOException { logger.error(GWConstants.LOG_S3OBJECT_OPERATION_FAILED_FILE_RENAME, srcFile.getAbsolutePath(), destFile.getAbsolutePath()); } } - - // public void setAttributeFileReplication(File file, String value, String replicaDiskID) { - // UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); - // try { - // view.write(GWConstants.FILE_ATTRIBUTE_REPLICATION, Charset.defaultCharset().encode(value)); - // view.write(GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID, Charset.defaultCharset().encode(replicaDiskID)); - // } catch (IOException e) { - // PrintStack.logging(logger, e); - // } - // } - - public String getAttributeFileReplication(File file) { - UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); - ByteBuffer buf = null; - try { - buf = ByteBuffer.allocate(view.size(GWConstants.FILE_ATTRIBUTE_REPLICATION)); - view.read(GWConstants.FILE_ATTRIBUTE_REPLICATION, buf); - buf.flip(); - } catch (IOException e) { - logger.error(e.getMessage()); - } catch (IllegalArgumentException iae) { - logger.error(iae.getMessage()); - } catch (SecurityException e) { - logger.error(e.getMessage()); - } - - return Charset.defaultCharset().decode(buf).toString(); - } - - public String getAttributeFileReplicaDiskID(File file) { - UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); - ByteBuffer buf = null; - try { - buf = ByteBuffer.allocate(view.size(GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID)); - view.read(GWConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID, buf); - buf.flip(); - } catch (IOException e) { - logger.error(e.getMessage()); - } catch (IllegalArgumentException iae) { - logger.error(iae.getMessage()); - } catch (SecurityException e) { - logger.error(e.getMessage()); - } - - return Charset.defaultCharset().decode(buf).toString(); - } } diff --git a/core/src/com/pspace/ifs/ksan/gw/object/S3ServerSideEncryption.java b/core/src/com/pspace/ifs/ksan/gw/object/S3ServerSideEncryption.java index 580b7ed6..28fd179b 100644 --- a/core/src/com/pspace/ifs/ksan/gw/object/S3ServerSideEncryption.java +++ b/core/src/com/pspace/ifs/ksan/gw/object/S3ServerSideEncryption.java @@ -39,6 +39,11 @@ public S3ServerSideEncryption(String encryptionXml, S3Metadata s3Metadata, S3Par this.encryptionXml = encryptionXml; } + public S3ServerSideEncryption(String encryptionXml, String serverSideEncryption, String customerAlgorithm, String customerKey, String customerKeyMD5, S3Parameter s3Parameter) { + super(serverSideEncryption, customerAlgorithm, customerKey, customerKeyMD5, s3Parameter); + this.encryptionXml = encryptionXml; + } + public void srcbuild() throws GWException { if (!Strings.isNullOrEmpty(customerAlgorithm) && customerAlgorithm.equalsIgnoreCase(GWConstants.AES256) == true) { if (!Strings.isNullOrEmpty(customerKey) && !Strings.isNullOrEmpty(customerKeyMD5)) { diff --git a/core/src/com/pspace/ifs/ksan/gw/object/objmanager/ObjManagers.java b/core/src/com/pspace/ifs/ksan/gw/object/objmanager/ObjManagers.java new file mode 100644 index 00000000..b820349b --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/object/objmanager/ObjManagers.java @@ -0,0 +1,78 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.object.objmanager; + +import java.io.IOException; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.utils.ObjectManagerConfig; +import com.pspace.ifs.ksan.objmanager.ObjManager; +import com.pspace.ifs.ksan.objmanager.ObjManagerConfig; +import com.pspace.ifs.ksan.gw.utils.GWConfig; + +public class ObjManagers { + private static final Logger logger = LoggerFactory.getLogger(ObjManagers.class); + private ObjManagerConfig config; + private ObjManager[] arrayObjManager = new ObjManager[(int)GWConfig.getInstance().getObjManagerCount()]; + private int index; + + public static ObjManagers getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static ObjManagers INSTANCE = new ObjManagers(); + } + + private ObjManagers() { + try { + config = new ObjManagerConfig(ObjectManagerConfig.getInstance().getDbRepository(), + ObjectManagerConfig.getInstance().getDbHost(), + ObjectManagerConfig.getInstance().getDbPort(), + ObjectManagerConfig.getInstance().getDbName(), + ObjectManagerConfig.getInstance().getDbUserName(), + ObjectManagerConfig.getInstance().getDbPassword(), + ObjectManagerConfig.getInstance().getMqHost(), + ObjectManagerConfig.getInstance().getMqUser(), + ObjectManagerConfig.getInstance().getMqPassword(), + (long)ObjectManagerConfig.getInstance().getMqPort(), + ObjectManagerConfig.getInstance().getMqQueueName(), + ObjectManagerConfig.getInstance().getMqExchangeName(), + ObjectManagerConfig.getInstance().getMqOsdExchangeName()); + } catch (Exception e) { + logger.error(e.getMessage()); + } + + index = -1; + } + + public void init() throws Exception { + for (int i = 0; i < 100; i++) { + arrayObjManager[i] = new ObjManager(config); + } + } + + public synchronized ObjManager getObjManager() throws Exception { + if (index >= 99) { + index = -1; + } + + index++; + + if (arrayObjManager[index] == null) { + arrayObjManager[index] = new ObjManager(config); + } + + return arrayObjManager[index]; + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/pom.xml b/core/src/com/pspace/ifs/ksan/gw/pom.xml index e510659e..d5e73f1b 100644 --- a/core/src/com/pspace/ifs/ksan/gw/pom.xml +++ b/core/src/com/pspace/ifs/ksan/gw/pom.xml @@ -13,6 +13,11 @@ + + com.github.seancfoley + ipaddress + 5.3.3 + com.pspace.ifs.ksan.libs ksan-libs @@ -99,6 +104,11 @@ org.apache.commons commons-dbcp2 2.9.0 + + + org.apache.commons + commons-pool2 + 2.11.1 com.zaxxer @@ -147,6 +157,11 @@ commons-crypto 1.1.0 + + de.sfuhrm + openssl4j + 0.2.0 + diff --git a/core/src/com/pspace/ifs/ksan/gw/sign/AzuSigning.java b/core/src/com/pspace/ifs/ksan/gw/sign/AzuSigning.java new file mode 100644 index 00000000..81490ce8 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/sign/AzuSigning.java @@ -0,0 +1,265 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.sign; + +import org.apache.commons.codec.binary.Base64; + +import java.net.URLDecoder; +import java.nio.charset.StandardCharsets; +import java.util.Collections; +import java.util.Iterator; +import java.util.Map; +import java.util.Random; +import java.util.SortedMap; +import java.util.TreeMap; + +import javax.crypto.Mac; +import javax.crypto.spec.SecretKeySpec; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import com.pspace.ifs.ksan.gw.identity.S3User; +import com.pspace.ifs.ksan.gw.data.azure.AzuDataRequest; +import com.pspace.ifs.ksan.gw.identity.AzuParameter; +import com.pspace.ifs.ksan.gw.utils.AzuConstants; +import com.pspace.ifs.ksan.gw.utils.S3UserManager; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.google.common.base.Strings; +import com.google.common.net.HttpHeaders; + +public class AzuSigning { + private AzuParameter parameter; + private String stringToSign; + private String contentEncoding; + private String contentLanguage; + private String contentLength; + private String contentMD5; + private String contentType; + private String date; + private String ifModifiedSince; + private String ifMatch; + private String ifNoneMatch; + private String ifUnmodifiedSince; + private String range; + SortedMap canonicalizedHeaderList = new TreeMap(); + SortedMap canonicalizedResourceList = new TreeMap(); + + private static Logger logger = LoggerFactory.getLogger(AzuSigning.class); + + public AzuSigning(AzuParameter parameter) throws Exception { + contentEncoding = ""; + contentLanguage = ""; + contentLength = ""; + contentMD5 = ""; + contentType = ""; + date = ""; + ifModifiedSince = ""; + ifMatch = ""; + ifNoneMatch = ""; + ifUnmodifiedSince = ""; + range = ""; + + this.parameter = parameter; + String authentication = parameter.getRequest().getHeader(HttpHeaders.AUTHORIZATION); + if (authentication == null) { + logger.info("does not have Authorization header ..."); + S3User user = S3UserManager.getInstance().getUserByName(parameter.getUserName()); + if (user == null) { + logger.error("User not found : {}", parameter.getUserName()); + } else { + parameter.setUser(user); + } + return; + } + logger.debug("authentication : {}", authentication); + String[] auths = authentication.split(" "); + String[] sharedKeys = auths[1].split(":"); + String account = sharedKeys[0]; + String key = sharedKeys[1]; + logger.debug("account : {}, key : {}", account, key); + + String x_ms_client_request_id = parameter.getRequest().getHeader(AzuConstants.X_MS_CLIENT_REQUEST_ID); + String x_ms_date = parameter.getRequest().getHeader(AzuConstants.X_MS_DATE); + String x_ms_version = parameter.getRequest().getHeader(AzuConstants.X_MS_VERSION); + + getHeaderStringToSign(); + + stringToSign = parameter.getMethod() + "\n" + + contentEncoding + "\n" // Content-Encoding + + contentLanguage + "\n" // Content-Language// Content-Language + + contentLength + "\n" // Content-Length + + contentMD5 + "\n" // content-MD5 + + contentType + "\n" // Content-Type + + date + "\n" // Date + + ifModifiedSince + "\n" // If-Modified-Since + + ifMatch + "\n" // If-Match + + ifNoneMatch + "\n" // If-None-Match + + ifUnmodifiedSince + "\n" // If-Unmodified-Since + + range + "\n" // Range + + getCanonicalizedHeaderString() + + getCanonicalizedResourceString(account); + + logger.debug("stringToSign : {}", stringToSign); + String authKey = getAuthenticationString(account, stringToSign); + if (authKey == null) { + throw new Exception(); + } + + if (key.equals(authKey)) { + logger.info("match key : {}, {}", key, authKey); + } else { + logger.info("does not match key : {}, {}", key, authKey); + throw new Exception(); + } + } + + private void getHeaderStringToSign() { + for (String headerName : Collections.list(parameter.getRequest().getHeaderNames())) { + for (String headerValue : Collections.list(parameter.getRequest().getHeaders(headerName))) { + if (headerValue != null) { + if (headerName.toLowerCase().startsWith(AzuConstants.CANONICAL_HEADER_START_WITH)) { + canonicalizedHeaderList.put(headerName, headerValue); + } else if (headerName.equals(HttpHeaders.CONTENT_ENCODING)) { + contentEncoding = Strings.nullToEmpty(headerValue); + } else if (headerName.equals(HttpHeaders.CONTENT_LANGUAGE)) { + contentLanguage = Strings.nullToEmpty(headerValue); + } else if (headerName.equals(HttpHeaders.CONTENT_LENGTH)) { + contentLength = Strings.nullToEmpty(headerValue); + if (contentLength.equals("0")) { + contentLength = ""; + } + } else if (headerName.equals(HttpHeaders.CONTENT_MD5)) { + contentMD5 = Strings.nullToEmpty(headerValue); + } else if (headerName.equals(HttpHeaders.CONTENT_TYPE)) { + contentType = Strings.nullToEmpty(headerValue); + } else if (headerName.equals(HttpHeaders.DATE)) { + date = Strings.nullToEmpty(headerValue); + } else if (headerName.equals(HttpHeaders.IF_MODIFIED_SINCE)) { + ifModifiedSince = Strings.nullToEmpty(headerValue); + } else if (headerName.equals(HttpHeaders.IF_MATCH)) { + ifMatch = Strings.nullToEmpty(headerValue); + } else if (headerName.equals(HttpHeaders.IF_NONE_MATCH)) { + ifNoneMatch = Strings.nullToEmpty(headerValue); + } else if (headerName.equals(HttpHeaders.IF_UNMODIFIED_SINCE)) { + ifUnmodifiedSince = Strings.nullToEmpty(headerValue); + } else if (headerName.equals(HttpHeaders.RANGE)) { + range = Strings.nullToEmpty(headerValue); + } + } + } + } + } + + private String getHeaderValueToSign(String headerName) { + String value = parameter.getRequest().getHeader(headerName); + if (Strings.isNullOrEmpty(value)) { + value = ""; + } + return value; + } + + private String getCanonicalizedHeaderString() { + String headers = ""; + for (Iterator> it = canonicalizedHeaderList.entrySet().iterator(); it.hasNext();) { + Map.Entry entry = it.next(); + headers += entry.getKey() + ":" + entry.getValue() + "\n"; + } + + return headers; + } + + private String getCanonicalizedResourceString(String account) { + String resource = "/" + account; + try { + String uri = parameter.getRequest().getRequestURI(); + uri = URLDecoder.decode(uri, AzuConstants.CHARSET_UTF_8); + resource += uri; + + if (!Strings.isNullOrEmpty(parameter.getComp())) { + if (!Strings.isNullOrEmpty(resource)) { + resource += "\n"; + } + resource += "comp:" + parameter.getComp(); + } + if (!Strings.isNullOrEmpty(parameter.getDelimiter())) { + if (!Strings.isNullOrEmpty(resource)) { + resource += "\n"; + } + resource += "delimiter:" + parameter.getDelimiter(); + } + if (!Strings.isNullOrEmpty(parameter.getInclude())) { + if (!Strings.isNullOrEmpty(resource)) { + resource += "\n"; + } + resource += "include:" + parameter.getInclude(); + } + if (!Strings.isNullOrEmpty(parameter.getMaxResults())) { + if (!Strings.isNullOrEmpty(resource)) { + resource += "\n"; + } + resource += "maxresults:" + parameter.getMaxResults(); + } + if (!Strings.isNullOrEmpty(parameter.getPrefix())) { + if (!Strings.isNullOrEmpty(resource)) { + resource += "\n"; + } + resource += "prefix:" + parameter.getPrefix(); + } + if (!Strings.isNullOrEmpty(parameter.getRestype())) { + if (!Strings.isNullOrEmpty(resource)) { + resource += "\n"; + } + resource += "restype:" + parameter.getRestype(); + } + if (!Strings.isNullOrEmpty(parameter.getTimeout())) { + if (!Strings.isNullOrEmpty(resource)) { + resource += "\n"; + } + resource += "timeout:" + parameter.getTimeout(); + } + } catch (Exception e) { + PrintStack.logging(logger, e); + } + + return resource; + } + + private String getAuthenticationString(String account, String stringToSign) { + try { + Mac mac = Mac.getInstance("HmacSHA256"); + // String key = AzuSharedKeyManager.getInstance().getSharedKey(account); + S3User user = S3UserManager.getInstance().getUserByName(account); + if (user == null) { + logger.error("User not found : {}", account); + return ""; + } + parameter.setUser(user); + String key = user.getAzureKey(); + if (key == null) { + return null; + } + + byte[] keyBytes = java.util.Base64.getDecoder().decode(key.getBytes()); + mac.init(new SecretKeySpec(keyBytes, "HmacSHA256")); + String authKey = Base64.encodeBase64String(mac.doFinal(stringToSign.getBytes("UTF-8"))); + return authKey; + } catch (Exception e) { + PrintStack.logging(logger, e); + } + + return null; + } + + public S3User getUser() { + return parameter.getUser(); + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/sign/ChunkedInputStream.java b/core/src/com/pspace/ifs/ksan/gw/sign/ChunkedInputStream.java index 16b0486a..56879da1 100644 --- a/core/src/com/pspace/ifs/ksan/gw/sign/ChunkedInputStream.java +++ b/core/src/com/pspace/ifs/ksan/gw/sign/ChunkedInputStream.java @@ -16,7 +16,8 @@ import com.google.common.io.ByteStreams; import com.pspace.ifs.ksan.gw.utils.GWConstants; - +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * Parse an AWS v4 signature chunked stream. Reference: @@ -32,7 +33,7 @@ public final class ChunkedInputStream extends FilterInputStream { value = GWConstants.LOG_CHUNKED_INPUT_STREAM_URF, justification = GWConstants.LOG_CHUNKED_JUSTIFICATION) private String currentSignature; - + Logger logger = LoggerFactory.getLogger(ChunkedInputStream.class); public ChunkedInputStream(InputStream is) { super(is); } @@ -69,6 +70,7 @@ public int read(byte[] b, int off, int len) throws IOException { } b[off + i] = (byte) ch; } + logger.info("chunkedInput ... read ..."); if (i == 0) { return -1; } diff --git a/core/src/com/pspace/ifs/ksan/gw/sign/S3Signature.java b/core/src/com/pspace/ifs/ksan/gw/sign/S3Signature.java index 8a37f328..b421e56c 100644 --- a/core/src/com/pspace/ifs/ksan/gw/sign/S3Signature.java +++ b/core/src/com/pspace/ifs/ksan/gw/sign/S3Signature.java @@ -43,6 +43,7 @@ import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -77,7 +78,9 @@ final public class S3Signature { GWConstants.PARAMETER_CORS, GWConstants.PARAMETER_TAGGING, GWConstants.PARAMETER_REPLICATION, - GWConstants.PARAMETER_DELETE + GWConstants.PARAMETER_DELETE, + GWConstants.PARAMETER_TAG_INDEX, + GWConstants.PARAMETER_RESTORE ); S3Signature() { @@ -113,30 +116,31 @@ public String createAuthorizationSignature( } // Build string to sign - StringBuilder builder = new StringBuilder() - .append(request.getMethod()) - .append(GWConstants.CHAR_NEWLINE) - .append(Strings.nullToEmpty(request.getHeader(HttpHeaders.CONTENT_MD5))) - .append(GWConstants.CHAR_NEWLINE) - .append(Strings.nullToEmpty(request.getHeader(HttpHeaders.CONTENT_TYPE))) - .append(GWConstants.CHAR_NEWLINE); + StringBuilder builder = new StringBuilder(); String expires = request.getParameter(GWConstants.EXPIRES); if (queryAuth) { + builder.append(request.getMethod()) + .append(GWConstants.CHAR_NEWLINE) + .append(Strings.nullToEmpty(request.getParameter(HttpHeaders.CONTENT_MD5))) + .append(GWConstants.CHAR_NEWLINE) + .append(Strings.nullToEmpty(request.getParameter(HttpHeaders.CONTENT_TYPE))) + .append(GWConstants.CHAR_NEWLINE); // If expires is not nil, then it is query string sign // If expires is nil,maybe alse query string sign // So should check other accessid para ,presign to judge. // not the expires - if(expires == null) { - builder.append(Strings.nullToEmpty(expires)); - } else { - logger.info(GWConstants.LOG_S3SIGNATURE_EXPIRES, expires); - builder.append(Strings.nullToEmpty(expires)); - } + builder.append(Strings.nullToEmpty(request.getParameter(HttpHeaders.EXPIRES))); } else { + builder.append(request.getMethod()) + .append(GWConstants.CHAR_NEWLINE) + .append(Strings.nullToEmpty(request.getHeader(HttpHeaders.CONTENT_MD5))) + .append(GWConstants.CHAR_NEWLINE) + .append(Strings.nullToEmpty(request.getHeader(HttpHeaders.CONTENT_TYPE))) + .append(GWConstants.CHAR_NEWLINE); if (!bothDateHeader) { if (canonicalizedHeaders.containsKey(GWConstants.X_AMZ_DATE_LOWER)) { - builder.append(""); + builder.append(GWConstants.EMPTY_STRING); } else { if (request.getHeader(HttpHeaders.DATE) != null) { logger.info(GWConstants.LOG_S3SIGNATURE_DATE, request.getHeader(HttpHeaders.DATE)); @@ -165,8 +169,7 @@ public String createAuthorizationSignature( builder.append(uri); char separator = GWConstants.CHAR_QUESTION; - List subresources = Collections.list( - request.getParameterNames()); + List subresources = Collections.list(request.getParameterNames()); Collections.sort(subresources); for (String subresource : subresources) { if (SIGNED_SUBRESOURCES.contains(subresource)) { @@ -187,13 +190,12 @@ public String createAuthorizationSignature( Mac mac; try { mac = Mac.getInstance(GWConstants.HMAC_SHA1); - mac.init(new SecretKeySpec(credential.getBytes( - StandardCharsets.UTF_8), GWConstants.HMAC_SHA1)); + mac.init(new SecretKeySpec(credential.getBytes(StandardCharsets.UTF_8), GWConstants.HMAC_SHA1)); } catch (InvalidKeyException | NoSuchAlgorithmException e) { throw new RuntimeException(e); } - return BaseEncoding.base64().encode(mac.doFinal( - stringToSign.getBytes(StandardCharsets.UTF_8))); + + return BaseEncoding.base64().encode(mac.doFinal(stringToSign.getBytes(StandardCharsets.UTF_8))); } private byte[] signMessage(byte[] data, byte[] key, String algorithm) @@ -341,7 +343,7 @@ private String createCanonicalRequest(HttpServletRequest request, } logger.info(canonicalRequest); - + return getMessageDigest( canonicalRequest.getBytes(StandardCharsets.UTF_8), hashAlgorithm); @@ -486,7 +488,7 @@ private String buildCanonicalHeadersXFF(HttpServletRequest request, } // 2021-04-14 UTF-8 signing fix - value = value.replaceAll(GWConstants.CHARSET_UTF_8, GWConstants.CHARSET_UTF_8_LOWER); + value = value.replaceAll(Constants.CHARSET_UTF_8, GWConstants.CHARSET_UTF_8_LOWER); values.add(value); } diff --git a/core/src/com/pspace/ifs/ksan/gw/sign/S3Signing.java b/core/src/com/pspace/ifs/ksan/gw/sign/S3Signing.java index a846b557..0b3be339 100644 --- a/core/src/com/pspace/ifs/ksan/gw/sign/S3Signing.java +++ b/core/src/com/pspace/ifs/ksan/gw/sign/S3Signing.java @@ -42,6 +42,7 @@ import com.pspace.ifs.ksan.gw.identity.S3User; import com.pspace.ifs.ksan.gw.object.objmanager.ObjManagerHelper; import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.gw.utils.S3UserManager; import com.pspace.ifs.ksan.gw.utils.GWConstants; import com.pspace.ifs.ksan.gw.utils.GWUtils; @@ -106,7 +107,7 @@ public S3Parameter publicvalidation() throws GWException { for (int i = 0; i < path.length; i++) { try { - path[i] = URLDecoder.decode(path[i], GWConstants.CHARSET_UTF_8); + path[i] = URLDecoder.decode(path[i], Constants.CHARSET_UTF_8); } catch (UnsupportedEncodingException e) { PrintStack.logging(logger, e); throw new GWException(GWErrorCode.SERVER_ERROR, GWConstants.LOG_S3SIGNING_UNSUPPORT_ENCODING_LANGUAGE, s3Parameter); @@ -194,7 +195,7 @@ public S3Parameter validation(boolean isAdmin) throws GWException { String[] path = uri.split(GWConstants.SLASH, 3); for (int i = 0; i < path.length; i++) { try { - path[i] = URLDecoder.decode(path[i], GWConstants.CHARSET_UTF_8); + path[i] = URLDecoder.decode(path[i], Constants.CHARSET_UTF_8); } catch (UnsupportedEncodingException e) { throw new GWException(GWErrorCode.SERVER_ERROR, GWConstants.LOG_S3SIGNING_UNSUPPORT_ENCODING_LANGUAGE, s3Parameter); } @@ -202,7 +203,8 @@ public S3Parameter validation(boolean isAdmin) throws GWException { S3AuthorizationHeader authHeader = null; String headerAuthorization = s3Parameter.getRequest().getHeader(HttpHeaders.AUTHORIZATION); - + boolean presignedUrl = false; + if (headerAuthorization == null) { String algorithm = s3Parameter.getRequest().getParameter(GWConstants.X_AMZ_ALGORITHM); @@ -215,6 +217,7 @@ public S3Parameter validation(boolean isAdmin) throws GWException { } headerAuthorization = GWConstants.AWS_SPACE + identity + GWConstants.COLON + signature; headernull = true; + presignedUrl = true; } else if (algorithm.equals(GWConstants.AWS4_HMAC_SHA256)) { //v4 query String credential = s3Parameter.getRequest().getParameter(GWConstants.X_AMZ_CREDENTIAL); String signedHeaders = s3Parameter.getRequest().getParameter(GWConstants.X_AMZ_SIGNEDHEADERS); @@ -228,11 +231,14 @@ public S3Parameter validation(boolean isAdmin) throws GWException { GWConstants.SIGN_REQEUEST_SIGNED_HEADERS + signedHeaders + GWConstants.SIGN_SIGNATURE + signature; headernull = true; + presignedUrl = true; } else { logger.error(GWConstants.LOG_S3SIGNING_UNKNOWN_ALGORITHM_VALUE, algorithm); throw new IllegalArgumentException(GWConstants.LOG_S3SIGNING_UNKNOWN_ALGORITHM + algorithm); } } + + logger.debug("headerAuthorization : {}", headerAuthorization); try { authHeader = new S3AuthorizationHeader(headerAuthorization); @@ -244,7 +250,7 @@ public S3Parameter validation(boolean isAdmin) throws GWException { String requestIdentity = authHeader.identity; - if(requestIdentity == null) { + if (requestIdentity == null) { logger.error(GWConstants.LOG_S3SIGNING_ACCESS_NULL); throw new GWException(GWErrorCode.INVALID_ACCESS_KEY_ID, s3Parameter); } @@ -268,39 +274,6 @@ public S3Parameter validation(boolean isAdmin) throws GWException { if (headernull) { headerAuthorization = null; } - - boolean presignedUrl = false; - - if (headerAuthorization == null) { - String algorithm = s3Parameter.getRequest().getParameter(GWConstants.X_AMZ_ALGORITHM); - - if (algorithm == null) { //v2 query - String identity = s3Parameter.getRequest().getParameter(GWConstants.AWS_ACCESS_KEY_ID); - String signature = s3Parameter.getRequest().getParameter(GWConstants.SIGNATURE); - if (identity == null || signature == null) { - logger.error(GWConstants.LOG_S3SIGNING_V2_SIGNATURE_NULL, uri); - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - headerAuthorization = GWConstants.AWS_SPACE + identity + GWConstants.COLON + signature; - presignedUrl = true; - } else if (algorithm.equals(GWConstants.AWS4_HMAC_SHA256)) { //v4 query - String credential = s3Parameter.getRequest().getParameter(GWConstants.X_AMZ_CREDENTIAL); - String signedHeaders = s3Parameter.getRequest().getParameter(GWConstants.X_AMZ_SIGNEDHEADERS); - String signature = s3Parameter.getRequest().getParameter(GWConstants.X_AMZ_SIGNATURE); - if (credential == null || signedHeaders == null || signature == null) { - logger.error(GWConstants.LOG_S3SIGNING_V4_CREDENTIAL_NULL, uri); - throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); - } - headerAuthorization = GWConstants.AWS4_HMAC_SHA256 + - GWConstants.SIGN_CREDENTIAL + credential + - GWConstants.SIGN_REQEUEST_SIGNED_HEADERS + signedHeaders + - GWConstants.SIGN_SIGNATURE + signature; - presignedUrl = true; - } else { - logger.error(GWConstants.LOG_S3SIGNING_UNKNOWN_ALGORITHM_VALUE, algorithm); - throw new IllegalArgumentException(GWConstants.LOG_S3SIGNING_UNKNOWN_ALGORITHM + algorithm); - } - } long dateSkew = 0; //date for timeskew check @@ -345,12 +318,6 @@ public S3Parameter validation(boolean isAdmin) throws GWException { } else { haveDate = false; } - - if (haveDate) { - GWUtils.isTimeSkewed(dateSkew, maxDateSkew, s3Parameter); - } - - String credential = user.getAccessSecret(); String expiresString = s3Parameter.getRequest().getParameter(GWConstants.EXPIRES); if (expiresString != null) { // v2 query @@ -376,6 +343,11 @@ public S3Parameter validation(boolean isAdmin) throws GWException { } } + if (haveDate) { + GWUtils.isTimeSkewed(dateSkew, maxDateSkew, s3Parameter); + } + + String credential = user.getAccessSecret(); String expectedSignature = null; // When presigned url is generated, it doesn't consider service path @@ -391,13 +363,14 @@ public S3Parameter validation(boolean isAdmin) throws GWException { } else { String contentSha256 = s3Parameter.getRequest().getHeader(GWConstants.X_AMZ_CONTENT_SHA256); byte[] payload = null; - int skip=0; + int skip = 0; if (s3Parameter.getRequest().getParameter(GWConstants.X_AMZ_ALGORITHM) != null) { payload = new byte[0]; } else if (GWConstants.STREAMING_AWS4_HMAC_SHA256_PAYLOAD.equals(contentSha256)) { payload = new byte[0]; s3Parameter.setInputStream(new ChunkedInputStream(s3Parameter.getInputStream())); + logger.info("chunked input stream ..."); } else if (GWConstants.UNSIGNED_PAYLOAD.equals(contentSha256)) { payload = new byte[0]; } else { @@ -453,11 +426,7 @@ public S3Parameter validation(boolean isAdmin) throws GWException { throw new GWException(GWErrorCode.SIGNATURE_DOES_NOT_MATCH, s3Parameter); } - if (s3Parameter.isAdmin()) { - s3Parameter.setUser(user); - } else { - s3Parameter.setUser(user); - } + s3Parameter.setUser(user); return s3Parameter; } diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/AsyncHandler.java b/core/src/com/pspace/ifs/ksan/gw/utils/AsyncHandler.java index 674f1a0b..edb8a5cc 100644 --- a/core/src/com/pspace/ifs/ksan/gw/utils/AsyncHandler.java +++ b/core/src/com/pspace/ifs/ksan/gw/utils/AsyncHandler.java @@ -27,13 +27,14 @@ public class AsyncHandler { public static CompletableFuture s3logging(S3Parameter s3Parameter) { CompletableFuture future = new CompletableFuture<>(); new Thread( () -> { - if (GWConfig.getInstance().isEventLog()) { - GWDB gwDB = GWUtils.getDBInstance(); - try { - gwDB.putS3logging(s3Parameter); - } catch (GWException e) { - PrintStack.logging(logger, e); - } + if (GWConfig.getInstance().isLogging()) { + // GWDB gwDB = GWUtils.getDBInstance(); + // try { + // gwDB.putS3logging(s3Parameter); + // } catch (GWException e) { + // PrintStack.logging(logger, e); + // } + GWLogging.getInstance().sendLog(s3Parameter); } }).start(); diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/AzuConfig.java b/core/src/com/pspace/ifs/ksan/gw/utils/AzuConfig.java new file mode 100644 index 00000000..19d5494b --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/utils/AzuConfig.java @@ -0,0 +1,51 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.utils; + +import java.io.FileInputStream; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.util.Properties; + +public class AzuConfig { + private Properties properties; + private String path; + public static final String DISK_PATH = "diskpath"; + + public static AzuConfig getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static AzuConfig INSTANCE = new AzuConfig(); + } + + private AzuConfig() { + properties = new Properties(); + try (InputStream config = new FileInputStream(AzuConstants.CONFIG_FILE)) { + properties.load(config); + } catch (FileNotFoundException e) { + throw new IllegalArgumentException("Config file is not exists."); + } catch (IOException e) { + throw new IllegalArgumentException("Config file load is failed."); + } + } + + public void configure() { + path = properties.getProperty(DISK_PATH); + } + + public String getDiskPath() { + return path; + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/AzuConstants.java b/core/src/com/pspace/ifs/ksan/gw/utils/AzuConstants.java new file mode 100644 index 00000000..47702269 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/utils/AzuConstants.java @@ -0,0 +1,239 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.utils; + +import java.io.File; + +public final class AzuConstants { + public static final String LOG_PRE_URI = "PREURI - {}"; + public static final String LOG_URI = "URI - {}"; + public static final String LOG_CLIENT_ADDRESS = "client address - {}"; + public static final String LOG_CLIENT_HOST = "client host - {}"; + public static final String LOG_METHOD = "method - {}"; + public static final String LOG_PARAMETER = "parameter - {}:{}"; + public static final String LOG_HEADER = "header - {}:{}"; + public static final String LOG_PATH = "path[{}] : {}"; + public static final String REQUEST_LOGS = "$logs"; + public static final String REQUEST_BLOCK_CHANGE_FEED = "$blobchangefeed"; + public static final String LOG_LOGS_BLOB_CHANGE_FEED = "request logs or blob change feed. container : {}, send SC_NOT_FOUND"; + + public static final String EMPTY_STRING = ""; + public static final String CONFIG_FILE = "/usr/local/ksan/etc/azu.conf"; + public static final String SHARED_KEY_AUTH_PATH = "/usr/local/ksan/etc/sharedKeys.info"; + + public static final String NEWLINE = "\n"; + public static final String TIME_FORMAT = "yyyy-MM-dd'T'HH:mm:ssXXX"; + public static final String REQUEST_ID = "RequestId"; + public static final String TIME = "Time"; + + public static final String CANONICAL_HEADER_START_WITH = "x-ms-"; + public static final String X_MS_CLIENT_REQUEST_ID = "x-ms-client-request-id"; + public static final String X_MS_RANGE = "x-ms-range"; + public static final String X_MS_DATE = "x-ms-date"; + public static final String X_MS_VERSION = "x-ms-version"; + public static final String X_MS_BLOB_CONTENT_TYPE = "X-Ms-Blob-Content-Type"; + public static final String X_MS_BLOB_CONTENT_MD5 = "X-Ms-Blob-Content-Md5"; + public static final String X_MS_BLOB_TYPE = "X-Ms-Blob-Type"; + public static final String X_MS_BLOB_CONTENT_ENCODING = "X-Ms-Blob-Content-Encoding"; + public static final String X_MS_BLOB_CONTENT_LANGUAGE = "X-Ms-Blob-Content-Language"; + public static final String X_MS_BLOB_CONTENT_DISPOSITION = "X-Ms-Blob-Content-Disposition"; + + public static final String PARAMETER_COMP = "comp"; + public static final String PARAMETER_RESTYPE = "restype"; + public static final String PARAMETER_DELIMITER = "delimiter"; + public static final String PARAMETER_MARKER = "marker"; + public static final String PARAMETER_PREFIX = "prefix"; + public static final String PARAMETER_INCLUDE = "include"; + public static final String PARAMETER_MAX_RESULTS = "maxresults"; + public static final String PARAMETER_TIMEOUT = "timeout"; + + public static final String HEADER_LASTMODIFIED = "last-modified"; + public static final String HEADER_X_MS_CREATION_TIME = "x-ms-creation-time"; + public static final String HEADER_X_MS_BLOB_TYPE = "x-ms-blob-type"; + public static final String HEADER_X_MS_LEASE_STATE = "x-ms-lease-state"; + public static final String HEADER_X_MS_LEASE_STATUS = "x-ms-lease-status"; + public static final String HEADER_ETAG = "etag"; + public static final String HEADER_CONTENT_MD5 = "content-md5"; + + public static final String BLOB_TYPE_BLOCKBLOB = "BlockBlob"; + public static final String LEASE_STATE_AVAILABLE = "available"; + public static final String LEASE_STATUS_UNLOCKED = "unlocked"; + public static final String ETAG_DEFAULT = "0x1DB589B70D48620"; + + public static final String HTTP_METHOD_GET = "GET"; + public static final String HTTP_METHOD_DELETE = "DELETE"; + public static final String HTTP_METHOD_HEAD = "HEAD"; + public static final String HTTP_METHOD_PUT = "PUT"; + + public static final String RESTYPE_SERVICE = "service"; + public static final String RESTYPE_CONTAINER = "container"; + public static final String RESTYPE_ACCOUNT = "account"; + + public static final String PATH_CATEGORY_ROOT = "root"; + public static final String PATH_CATEGORY_CONTAINER = "container"; + public static final String PATH_CATEGORY_BLOB = "blob"; + + public static final String COMP_BLOCK = "block"; + public static final String COMP_BLOCKLIST = "blocklist"; + public static final String COMP_PROPERTIES = "properties"; + + public static final String CONTAINER_DIRECTORY = "__container__"; + public static final String BLOB_INFO_DIRECTORY = "__info__"; + public static final String BLOB_BLOCK_DIRECTORY = "__block__"; + + public static final String MD5 = "MD5"; + public static final int MAXBUFSIZE = 524288; + public static final int BUFSIZE = 262144; + + public static final String TIMEZONE_GMT = "GMT"; + public static final String TIME_GMT_FORMAT = "E, dd MMM yyyy HH:mm:ss z"; + public static final String LOCALE_EN = "en"; + public static final String CHARSET_UTF_8 = "UTF-8"; + + public static final String COLON = ":"; + public static final String SEPARATOR = "/"; + public static final char SEPARATOR_CHAR = '/'; + + public static final String NULL = "null"; + + public static final String CONTENT_TYPE_XML = "application/xml"; + public static final String CONTENT_TYPE_TEXT = "text/plain"; + public static final String CONTENT_TYPE_APPLICATION = "application/octet-stream"; + + // BlockList + public static final String BLOCK_LIST = "Blocklist"; + public static final String BLOCK_LATEST = "Latest"; + + // XML Element Name + public static final String XML_ELEMENT_LOGGING = "Logging"; + public static final String XML_ELEMENT_VERSION = "Version"; + public static final String XML_ELEMENT_DELETE = "Delete"; + public static final String XML_ELEMENT_READ = "Read"; + public static final String XML_ELEMENT_WRITE = "Write"; + public static final String XML_ELEMENT_RETENTION_POLICY = "RetentionPolicy"; + public static final String XML_ELEMENT_ENABLED = "Enabled"; + public static final String XML_ELEMENT_HOUR_METRICS = "HourMetrics"; + public static final String XML_ELEMENT_MINUTE_METRICS = "MinuteMetrics"; + public static final String XML_ELEMENT_CORS = "Cors"; + public static final String XML_ELEMENT_DEFAULT_SERVICE_VERSION = "DefaultServiceVersion"; + public static final String XML_ELEMENT_STATIC_WEBSITE = "StaticWebsite"; + + public static final String XML_ELEMENT_ENUMERATION_RESULT = "EnumerationResults"; + public static final String XML_ATTRIBUTE_SERVICE_ENDPOINT = "ServiceEndpoint"; + public static final String XML_ATTRIBUTE_SERVICE_ENDPOINT_VALUE = "http://127.0.0.1:10000/"; + public static final String XML_ATTRIBUTE_CONTAINER_NAME = "ContainerName"; + public static final String XML_ELEMENT_PREFIX = "Prefix"; + public static final String XML_ELEMENT_MARKER = "Marker"; + public static final String XML_ELEMENT_DELIMITER = "Delimiter"; + public static final String XML_ELEMENT_MAX_RESULT = "MaxResults"; + public static final String XML_ELEMENT_LISTBLOB_MAX_RESULT_VALUE = "1000"; + public static final String XML_ELEMENT_LISTCONTAINER_MAX_RESULT_VALUE = "5000"; + public static final String XML_ELEMENT_CONTAINERS = "Containers"; + public static final String XML_ELEMENT_CONTAINER = "Container"; + public static final String XML_ELEMENT_BLOBS = "Blobs"; + public static final String XML_ELEMENT_BLOB = "Blob"; + public static final String XML_ELEMENT_NAME = "Name"; + public static final String XML_ELEMENT_PROPERTIES = "Properties"; + public static final String XML_ELEMENT_LAST_MODIFIED = "Last-Modified"; + public static final String XML_ELEMENT_ETAG = "Etag"; + public static final String XML_ELEMENT_LEASE_STATUS = "LeaseStatus"; + public static final String XML_ELEMENT_LEASE_STATE = "LeaseState"; + public static final String XML_ELEMENT_HAS_IMMUTABILITY_POLICY = "HasImmutabilityPolicy"; + public static final String XML_ELEMENT_HAS_LEGALHOLD = "HasLegalHold"; + public static final String XML_ELEMENT_NEXT_MARKER = "NextMarker"; + public static final String XML_ELEMENT_CREATION_TIME = "Creation-Time"; + public static final String XML_ELEMENT_CONTENT_LENGTH = "Content-Length"; + public static final String XML_ELEMENT_CONTENT_TYPE = "Content-Type"; + public static final String XML_ELEMENT_CONTENT_ENCODING = "Content-Encoding"; + public static final String XML_ELEMENT_CONTENT_LANGUAGE = "Content-Language"; + public static final String XML_ELEMENT_CONTENT_MD5 = "Content-MD5"; + public static final String XML_ELEMENT_CONTENT_DISPOSITION = "Content-Disposition"; + public static final String XML_ELEMENT_CACHE_CONTROL = "Cache-Control"; + public static final String XML_ELEMENT_BLOB_TYPE = "BlobType"; + public static final String XML_ELEMENT_SERVER_ENCRYPTED = "ServerEncrypted"; + public static final String XML_ELEMENT_ACCESS_TIER = "AccessTier"; + public static final String XML_ELEMENT_ACCESS_TIER_INFERRED = "AccessTierInferred"; + public static final String XML_ELEMENT_ACCESS_TIER_CHANGE_TIME = "AccessTierChangeTime"; + public static final String XML_ELEMENT_BLOB_PREFIX = "BlobPrefix"; + public static final String XML_ELEMENT_VALUE_FALSE = "false"; + public static final String XML_ELEMENT_VALUE_TRUE = "true"; + public static final String XML_ELEMENT_VALUE_HOT = "Hot"; + public static final String XML_ELEMENT_VALUE_ETAG = "0x22178A0E1864500"; + + public static final String XML_ELEMENT_ERROR = "Error"; + public static final String XML_ELEMENT_CODE = "Code"; + public static final String XML_ELEMENT_MESSAGE = "Message"; + + public static final String XML_VERSION = ""; + + public static final String BLOCKID = "blockid"; + public static final String X_MS_BLOCK_CONTENT_MD5 = "X-Ms-Blob-Content-Md5"; + + public static final String LATEST = "Latest"; + + // StorageServiceProperties + public static final String STORAGE_SERVICE_PROPERTIES = "StorageServiceProperties"; + + // AzuRequest + public static final String LOG_CONTAINER_IS_NOT_EXIST = "container is not exist : {}"; + + // CreateContainer + public static final String LOG_CREATE_CONTAINER_START = "CreateContainer ..."; + public static final String LOG_CREATE_CONTAINER_NAME = "container : {}"; + + // HeadContainer + public static final String LOG_HAED_CONTAINER_START = "HeadContainer ..."; + + // DeleteContainer + public static final String LOG_DELETE_CONTAINER_START = "DeleteContainer ..."; + + // ListContainer + public static final String LOG_LIST_CONTAINER_START = "ListContainer ..."; + public static final String LOG_LIST_CONTAINER_NAME_DATE = "container : {}, date : {}"; + + // CreateBlob + public static final String LOG_CREATE_BLOB_START = "CreateBlob ..."; + public static final String LOG_CREATE_BLOB_PRIMARY_DISK_ID = "obj prmary disk id : {}"; + + // HeadBlob + public static final String LOG_HEAD_BLOB_START = "HeadBlob ..."; + + // GetBlob + public static final String LOG_GET_BLOB_START = "GetBlob ..."; + + // DeleteBlob + public static final String LOG_DELETE_BLOB_START = "DeleteBlob ..."; + + // ListBlob + public static final String LOG_LIST_BLOB_START = "ListBlob ..."; + + + // GetProperties + public static final String LOG_GET_PROPERTIES_START = "GetProperties ..."; + public static final String VERSION_VALUE = "1.0"; + public static final String SERVICE_VERSION_VALUE = "2021-10-04"; + + // UploadBlock + public static final String LOG_UPLOAD_BLOCK_START = "UploadBlock ..."; + + // CompleteBlockList + public static final String LOG_COMPLETE_BLOCK_LIST_START = "CompleteBlockList ..."; + + // SharedKeyAuth + public static final String LOG_SHARED_KEY_AUTH_START = "SharedKeyAuth ..."; + public static final String INVALID_STORAGE_ACCOUNT = "Invalid storage account."; + public static final String INVALID_OPERATION = "InvalidOperation"; + + // AzuObjectOperation + public static final String LOG_S3OBJECT_OPERATION_DELETE = "delete {}, {}"; +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/AzuUtils.java b/core/src/com/pspace/ifs/ksan/gw/utils/AzuUtils.java new file mode 100644 index 00000000..523759cc --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/utils/AzuUtils.java @@ -0,0 +1,44 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.utils; + +import java.io.File; + +public class AzuUtils { + public static String getPath() { + return AzuConfig.getInstance().getDiskPath() + File.separator + AzuConstants.CONTAINER_DIRECTORY; + } + + public static String getPath(String path) { + return AzuConfig.getInstance().getDiskPath() + File.separator + AzuConstants.CONTAINER_DIRECTORY + File.separator + path; + } + + public static String getPath(String container, String blob) { + return AzuConfig.getInstance().getDiskPath() + File.separator + AzuConstants.CONTAINER_DIRECTORY + File.separator + container + File.separator + blob; + } + + public static String getInfoPath() { + return AzuConfig.getInstance().getDiskPath() + File.separator + AzuConstants.BLOB_INFO_DIRECTORY; + } + + public static String getInfoPath(String path) { + return AzuConfig.getInstance().getDiskPath() + File.separator + AzuConstants.BLOB_INFO_DIRECTORY + File.separator + path; + } + + public static String getInfoPath(String container, String blob) { + return AzuConfig.getInstance().getDiskPath() + File.separator + AzuConstants.BLOB_INFO_DIRECTORY + File.separator + container + File.separator + blob; + } + + public static String getBlockPath(String container, String blob, String id) { + return AzuConfig.getInstance().getDiskPath() + File.separator + AzuConstants.BLOB_BLOCK_DIRECTORY + File.separator + container + File.separator + blob + File.separator + id; + } +} + diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/GWConfig.java b/core/src/com/pspace/ifs/ksan/gw/utils/GWConfig.java index 81e5a08b..11a1f015 100644 --- a/core/src/com/pspace/ifs/ksan/gw/utils/GWConfig.java +++ b/core/src/com/pspace/ifs/ksan/gw/utils/GWConfig.java @@ -27,7 +27,7 @@ public class GWConfig { private String version; - private String dbRepository; + // private String dbRepository; // private long replicaCount; private String cacheDiskpath; private boolean isCacheDiskpath; @@ -47,20 +47,24 @@ public class GWConfig { private long osdPort = GWConstants.DEFAULT_OSD_PORT; private long osdClientCount = GWConstants.DEFAULT_OSD_CLIENT_SIZE; private long objManagerCount = GWConstants.DEFAULT_OBJMANAGER_SIZE; - private String dbHost; - private String database; - private long dbPort; - private String dbUser; - private String dbPass; - private long dbPoolSize; + // private String dbHost; + // private String database; + // private long dbPort; + // private String dbUser; + // private String dbPass; + // private long dbPoolSize; private boolean isNoOption; private boolean isNoIO; private boolean isNoDisk; private boolean isNoReplica; + private boolean isNoOperation; + private boolean isPREV; + private boolean isDBOP; + private boolean isS3OP; - private String eventLog; - private boolean isEventLog; + private String logging; + private boolean isLogging; private static final String VERSION = "version"; private static final String AUTHORIZATION = "gw.authorization"; @@ -78,16 +82,16 @@ public class GWConfig { private static final String OSD_CLIENT_COUNT = "gw.osd_client_count"; private static final String OBJMANAGER_COUNT = "gw.objmanager_count"; private static final String PERFORMANCE_MODE = "gw.performance_mode"; - private static final String DB_REPOSITORY = "gw.db_repository"; - private static final String DB_HOST = "gw.db_host"; - private static final String DB_NAME = "gw.db_name"; - private static final String DB_PORT = "gw.db_port"; - private static final String DB_USER = "gw.db_user"; - private static final String DB_PASSWORD = "gw.db_password"; - private static final String DB_POOL_SIZE = "gw.db_pool_size"; + // private static final String DB_REPOSITORY = "gw.db_repository"; + // private static final String DB_HOST = "gw.db_host"; + // private static final String DB_NAME = "gw.db_name"; + // private static final String DB_PORT = "gw.db_port"; + // private static final String DB_USER = "gw.db_user"; + // private static final String DB_PASSWORD = "gw.db_password"; + // private static final String DB_POOL_SIZE = "gw.db_pool_size"; private static final String CACHE_PATH = "gw.cache_diskpath"; // private static final String CACHE_FILE_SIZE = "gw.cache_file_size"; - private static final String EVENT_LOG = "gw.eventlog"; + private static final String EVENT_LOG = "gw.logging"; private static final String EQUAL = "="; private static final String ON = "on"; @@ -114,13 +118,13 @@ public void setVersion(String version) { this.version = version; } - public String getDbRepository() { - return dbRepository; - } + // public String getDbRepository() { + // return dbRepository; + // } - public void setDbRepository(String dbRepository) { - this.dbRepository = dbRepository; - } + // public void setDbRepository(String dbRepository) { + // this.dbRepository = dbRepository; + // } // public long getReplicaCount() { // return replicaCount; @@ -266,53 +270,53 @@ public void setObjManagerCount(long objManagerCount) { this.objManagerCount = objManagerCount; } - public String getDbHost() { - return dbHost; - } + // public String getDbHost() { + // return dbHost; + // } - public void setDbHost(String dbHost) { - this.dbHost = dbHost; - } + // public void setDbHost(String dbHost) { + // this.dbHost = dbHost; + // } - public String getDatabase() { - return database; - } + // public String getDatabase() { + // return database; + // } - public void setDatabase(String database) { - this.database = database; - } + // public void setDatabase(String database) { + // this.database = database; + // } - public long getDbPort() { - return dbPort; - } + // public long getDbPort() { + // return dbPort; + // } - public void setDbPort(long dbPort) { - this.dbPort = dbPort; - } + // public void setDbPort(long dbPort) { + // this.dbPort = dbPort; + // } - public String getDbUser() { - return dbUser; - } + // public String getDbUser() { + // return dbUser; + // } - public void setDbUser(String dbUser) { - this.dbUser = dbUser; - } + // public void setDbUser(String dbUser) { + // this.dbUser = dbUser; + // } - public String getDbPass() { - return dbPass; - } + // public String getDbPass() { + // return dbPass; + // } - public void setDbPass(String dbPass) { - this.dbPass = dbPass; - } + // public void setDbPass(String dbPass) { + // this.dbPass = dbPass; + // } - public long getDbPoolSize() { - return dbPoolSize; - } + // public long getDbPoolSize() { + // return dbPoolSize; + // } - public void setDbPoolSize(long dbPoolSize) { - this.dbPoolSize = dbPoolSize; - } + // public void setDbPoolSize(long dbPoolSize) { + // this.dbPoolSize = dbPoolSize; + // } public boolean isNoOption() { return isNoOption; @@ -346,8 +350,40 @@ public void setNoReplica(boolean isNoReplica) { this.isNoReplica = isNoReplica; } - public boolean isEventLog() { - return isEventLog; + public boolean isNoOperation() { + return isNoOperation; + } + + public void setNoOperation(boolean isNoOperation) { + this.isNoOperation = isNoOperation; + } + + public boolean isPREV() { + return isPREV; + } + + public void setPREV(boolean isPREV) { + this.isPREV = isPREV; + } + + public boolean isDBOP() { + return isDBOP; + } + + public void setDBOP(boolean isDBOP) { + this.isDBOP = isDBOP; + } + + public boolean isS3OP() { + return isS3OP; + } + + public void setS3OP(boolean isS3OP) { + this.isS3OP = isS3OP; + } + + public boolean isLogging() { + return isLogging; } public void setConfig(JSONObject jsonConfig) throws URISyntaxException { @@ -387,6 +423,10 @@ public void setConfig(JSONObject jsonConfig) throws URISyntaxException { setNoIO(false); setNoDisk(false); setNoReplica(false); + setNoOperation(false); + setPREV(false); + setDBOP(false); + setS3OP(false); } else { if (getPerformanceMode().equals(GWConstants.PERFORMANCE_MODE_NO_OPTION)) { setPerformanceMode(GWConstants.PERFORMANCE_MODE_NO_OPTION); @@ -394,39 +434,91 @@ public void setConfig(JSONObject jsonConfig) throws URISyntaxException { setNoIO(false); setNoDisk(false); setNoReplica(false); + setNoOperation(false); + setPREV(false); + setDBOP(false); + setS3OP(false); } else if (getPerformanceMode().equals(GWConstants.PERFORMANCE_MODE_NO_IO)) { setNoOption(false); setNoIO(true); setNoDisk(false); setNoReplica(false); + setNoOperation(false); + setPREV(false); + setDBOP(false); + setS3OP(false); } else if (getPerformanceMode().equals(GWConstants.PERFORMANCE_MODE_NO_DISK)) { setNoOption(false); setNoIO(false); setNoDisk(true); setNoReplica(false); + setNoOperation(false); + setPREV(false); + setDBOP(false); + setS3OP(false); } else if (getPerformanceMode().equals(GWConstants.PERFORMANCE_MODE_NO_REPLICA)) { setNoOption(false); setNoIO(false); setNoDisk(false); setNoReplica(true); - } + setNoOperation(false); + setPREV(false); + setDBOP(false); + setS3OP(false); + } else if (getPerformanceMode().equals(GWConstants.PERFORMANCE_MODE_NO_OPERATION)) { + setNoOption(false); + setNoIO(false); + setNoDisk(false); + setNoReplica(false); + setNoOperation(true); + setPREV(false); + setDBOP(false); + setS3OP(false); + } else if (getPerformanceMode().equals(GWConstants.PERFORMANCE_MODE_PREV)) { + setNoOption(false); + setNoIO(false); + setNoDisk(false); + setNoReplica(false); + setNoOperation(false); + setPREV(true); + setDBOP(false); + setS3OP(false); + } else if (getPerformanceMode().equals(GWConstants.PERFORMANCE_MODE_DB_OP)) { + setNoOption(false); + setNoIO(false); + setNoDisk(false); + setNoReplica(false); + setNoOperation(false); + setPREV(false); + setDBOP(true); + setS3OP(false); + } else if (getPerformanceMode().equals(GWConstants.PERFORMANCE_MODE_S3_OPERATION)) { + setNoOption(false); + setNoIO(false); + setNoDisk(false); + setNoReplica(false); + setNoOperation(false); + setPREV(false); + setDBOP(false); + setS3OP(true); + } } - eventLog = (String)jsonConfig.get(EVENT_LOG); - if (!Strings.isNullOrEmpty(eventLog) && eventLog.equalsIgnoreCase(OFF)) { - isEventLog = false; + logging = (String)jsonConfig.get(EVENT_LOG); + if (!Strings.isNullOrEmpty(logging) && logging.equalsIgnoreCase(OFF)) { + isLogging = false; } else { - eventLog = ON; - isEventLog = true; + logging = ON; + isLogging = true; } - setDbRepository((String)jsonConfig.get(DB_REPOSITORY)); - setDbHost((String)jsonConfig.get(DB_HOST)); - setDatabase((String)jsonConfig.get(DB_NAME)); - setDbPort((long)jsonConfig.get(DB_PORT)); - setDbUser((String)jsonConfig.get(DB_USER)); - setDbPass((String)jsonConfig.get(DB_PASSWORD)); - setDbPoolSize((long)jsonConfig.get(DB_POOL_SIZE)); + // setDbRepository((String)jsonConfig.get(DB_REPOSITORY)); + // setDbHost((String)jsonConfig.get(DB_HOST)); + // setDatabase((String)jsonConfig.get(DB_NAME)); + // setDbPort((long)jsonConfig.get(DB_PORT)); + // setDbUser((String)jsonConfig.get(DB_USER)); + // setDbPass((String)jsonConfig.get(DB_PASSWORD)); + // setDbPoolSize((long)jsonConfig.get(DB_POOL_SIZE)); setCacheDiskpath((String)jsonConfig.get(CACHE_PATH)); if (!Strings.isNullOrEmpty(getCacheDiskpath())) { @@ -452,13 +544,13 @@ public void setConfig(JSONObject jsonConfig) throws URISyntaxException { logger.debug("{}", getOsdClientCount()); logger.debug("{}", getObjManagerCount()); logger.debug(getPerformanceMode()); - logger.debug(getDbRepository()); - logger.debug(getDbHost()); - logger.debug(getDatabase()); - logger.debug("{}", getDbPort()); - logger.debug(getDbUser()); - logger.debug(getDbPass()); - logger.debug("{}", getDbPoolSize()); + // logger.debug(getDbRepository()); + // logger.debug(getDbHost()); + // logger.debug(getDatabase()); + // logger.debug("{}", getDbPort()); + // logger.debug(getDbUser()); + // logger.debug(getDbPass()); + // logger.debug("{}", getDbPoolSize()); logger.debug(getCacheDiskpath()); // logger.debug("{}", getCacheFileSize()); } @@ -476,20 +568,20 @@ public void saveConfigFile() throws IOException { fileWriter.write(MAX_FILE_SIZE + EQUAL + maxFileSize + "\n"); fileWriter.write(MAX_LIST_SIZE + EQUAL + maxListSize + "\n"); fileWriter.write(MAX_TIMESKEW + EQUAL + maxTimeSkew + "\n"); - fileWriter.write(EVENT_LOG + EQUAL + eventLog + "\n"); + fileWriter.write(EVENT_LOG + EQUAL + logging + "\n"); // fileWriter.write(REPLICATION + EQUAL + replicaCount + "\n"); fileWriter.write(OSD_PORT + EQUAL + osdPort + "\n"); fileWriter.write(JETTY_MAX_THREADS + EQUAL + jettyMaxThreads + "\n"); fileWriter.write(OSD_CLIENT_COUNT + EQUAL + osdClientCount + "\n"); fileWriter.write(OBJMANAGER_COUNT + EQUAL + objManagerCount + "\n"); fileWriter.write(PERFORMANCE_MODE + EQUAL + performanceMode + "\n"); - fileWriter.write(DB_REPOSITORY + EQUAL + dbRepository + "\n"); - fileWriter.write(DB_HOST + EQUAL + dbHost + "\n"); - fileWriter.write(DB_NAME + EQUAL + database + "\n"); - fileWriter.write(DB_PORT + EQUAL + dbPort + "\n"); - fileWriter.write(DB_USER + EQUAL + dbUser + "\n"); - fileWriter.write(DB_PASSWORD + EQUAL + dbPass + "\n"); - fileWriter.write(DB_POOL_SIZE + EQUAL + dbPoolSize + "\n"); + // fileWriter.write(DB_REPOSITORY + EQUAL + dbRepository + "\n"); + // fileWriter.write(DB_HOST + EQUAL + dbHost + "\n"); + // fileWriter.write(DB_NAME + EQUAL + database + "\n"); + // fileWriter.write(DB_PORT + EQUAL + dbPort + "\n"); + // fileWriter.write(DB_USER + EQUAL + dbUser + "\n"); + // fileWriter.write(DB_PASSWORD + EQUAL + dbPass + "\n"); + // fileWriter.write(DB_POOL_SIZE + EQUAL + dbPoolSize + "\n"); fileWriter.write(CACHE_PATH + EQUAL + cacheDiskpath + "\n"); // fileWriter.write(CACHE_FILE_SIZE + EQUAL + cacheFileSize + "\n"); fileWriter.close(); diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/GWConstants.java b/core/src/com/pspace/ifs/ksan/gw/utils/GWConstants.java index f051bfe5..1e3a2eae 100644 --- a/core/src/com/pspace/ifs/ksan/gw/utils/GWConstants.java +++ b/core/src/com/pspace/ifs/ksan/gw/utils/GWConstants.java @@ -33,6 +33,7 @@ public final class GWConstants { public static final String MQUEUE_NAME = "disk"; public static final String MQUEUE_EXCHANGE_NAME = "ksan.system"; public static final String MQUEUE_OSD_EXCHANGE_NAME = "OSDExchange"; + public static final String MQUEUE_LOG_EXCHANGE_NAME = "ksan.log"; public static final String MQUEUE_NAME_GW_CONFIG = "ksan-gw-configure-"; public static final String MQUEUE_NAME_GW_CONFIG_ROUTING_KEY = "*.services.gw.config.*"; @@ -50,6 +51,7 @@ public final class GWConstants { public static final String MQUEUE_NAME_GW_SERVICE_ADDED_ROUTING_KEY = "*.services.added"; public static final String MQUEUE_NAME_GW_SERVICE_UPDATED_ROUTING_KEY = "*.services.updated"; public static final String MQUEUE_NAME_GW_SERVICE_REMOVED_ROUTING_KEY = "*.services.removed"; + public static final String MQUEUE_NAME_GW_LOG_ADD = "*.services.gw.log.add"; public static final String PORTAL_REST_API_CONFIG_GW = "/api/v1/Config/KsanGw"; public static final String PORTAL_REST_API_DISKPOOLS_DETAILS = "/api/v1/DiskPools/Details"; @@ -74,6 +76,11 @@ public final class GWConstants { public static final String PERFORMANCE_MODE_NO_IO = "NO_IO"; public static final String PERFORMANCE_MODE_NO_DISK = "NO_DISK"; public static final String PERFORMANCE_MODE_NO_REPLICA = "NO_REPLICA"; + public static final String PERFORMANCE_MODE_NO_OPERATION = "NO_OPERATION"; + public static final String PERFORMANCE_MODE_PREV = "PREV"; + public static final String PERFORMANCE_MODE_DB_OP = "DB_OPERATION"; + public static final String PERFORMANCE_MODE_S3_OPERATION = "S3_OPERATION"; + public static final int JETTY_MAX_THREADS = 1000; public static final int JETTY_MAX_IDLE_TIMEOUT = 30000; public static final long MAX_FILE_SIZE = 100 * 1024 * 1024 * 1024; @@ -137,12 +144,6 @@ public final class GWConstants { public static final String JDBC_DRIVER = "org.mariadb.jdbc.Driver"; public static final String MARIADB_VALIDATION_QUERY = "select 1"; - public static final String OBJ_DIR = "obj"; - public static final String TEMP_DIR = "temp"; - public static final String TEMP_COMPLETE_DIR = "temp/complete"; - public static final String TEMP_COPY_DIR = "temp/copy"; - public static final String TRASH_DIR = "trash"; - public static final String EC_DIR = "ec"; public static final int RETRY_COUNT = 3; public static final String METHOD_PUT = "PUT"; @@ -155,6 +156,7 @@ public final class GWConstants { public static final String STRING_TRUE = "TRUE"; public static final String STRING_FALSE = "FALSE"; public static final String ALLOW = "Allow"; + public static final String DENY = "Deny"; public static final String STRING_ERROR = "Error"; public static final String CODE = "Code"; public static final String MESSAGE = "Message"; @@ -237,6 +239,8 @@ public final class GWConstants { public static final String VERSIONING_DISABLE_TAIL = "null"; public static final String GOVERNANCE = "GOVERNANCE"; public static final String COMPLIANCE = "COMPLIANCE"; + public static final String ON = "ON"; + public static final String OFF = "OFF"; public static final String S3_ARN = "arn:aws:s3"; public static final String HTTP = "http://"; @@ -260,6 +264,7 @@ public final class GWConstants { public static final String PARAMETER_LIFECYCLE = "lifecycle"; public static final String PARAMETER_PUBLIC_ACCESS_BLOCK = "publicAccessBlock"; public static final String PARAMETER_TAGGING = "tagging"; + public static final String PARAMETER_TAG_INDEX = "tag-index"; public static final String PARAMETER_ENCRYPTION = "encryption"; public static final String PARAMETER_OBJECT_LOCK = "object-lock"; public static final String PARAMETER_REPLICATION = "replication"; @@ -273,6 +278,7 @@ public final class GWConstants { public static final String PARAMETER_VERSIONING = "versioning"; public static final String PARAMETER_LIST_TYPE = "list-type"; public static final String PARAMETER_VERSIONS = "versions"; + public static final String PARAMETER_LIST_TAG_SEARCH = "list-tag-search"; public static final String PARAMETER_RETENTION = "retention"; public static final String PARAMETER_LEGAL_HOLD = "legal-hold"; public static final String PARAMETER_DELETE = "delete"; @@ -281,6 +287,7 @@ public final class GWConstants { public static final String PARAMETER_COPY_SOURCE = "x-amz-copy-source"; public static final String PARAMETER_TORRENT = "torrent"; public static final String PARAMETER_VERSION_ID = "versionId"; + public static final String PARAMETER_RESTORE = "restore"; public static final String SUB_PARAMETER_VERSIONID = "?versionId="; public static final String PARAMETER_BACKSLASH_VERSIONID = "\\?versionId="; public static final String CONTENT_TYPE_POST_OBJECT = "multipart/form-data; boundary="; @@ -355,6 +362,30 @@ public final class GWConstants { public static final String ACTION_BYPASS_GOVERNANCE_RETENTION = "s3:BypassGovernanceRetention"; + // policy condition keys constatns + public static final String KEY_AUTH_TYPE = "s3:authType"; + public static final String KEY_DELIMITER = "s3:delimiter"; + public static final String KEY_MAXKYES = "s3:max-keys"; + public static final String KEY_PREFIX = "s3:prefix"; + public static final String KEY_X_AMZ_CONTENT_SHA256 = "s3:x-amz-content-sha256"; + public static final String KEY_EXISTING_OBJECT_TAG = "s3:ExistingObjectTag"; + + public static final String KEY_X_AMZ_ACL = "s3:x-amz-acl"; + public static final String KEY_X_AMZ_COPY_SOURCE = "s3:x-amz-copy-source"; + public static final String KEY_X_AMZ_GRANT_FULL_CONTROL = "s3:x-amz-grant-full-control"; + public static final String KEY_XAMZ_GRANT_READ = "s3:x-amz-grant-read"; + public static final String KEY_X_AMZ_GRANT_READ_ACP = "s3:x-amz-grant-read-acp"; + public static final String KEY_X_AMZ_GRANT_WRITE = "s3:x-amz-grant-write"; + public static final String KEY_X_AMZ_GRANT_WRITE_ACP = "s3:x-amz-grant-write-acp"; + public static final String KEY_X_AMZ_METADATA_DIRECTIVE = "s3:x-amz-metadata-directive"; + public static final String KEY_X_AMZ_SERVER_SIDE_ENCRYPTION = "s3:x-amz-server-side-encryption"; + public static final String KEY_X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID = "s3:x-amz-server-side-encryption-aws-kms-key-id"; + public static final String KEY_X_AMZ_STORAGE_CLASS = "s3:x-amz-storage-class"; + public static final String KEY_X_AMZ_WEBSITE_REDIRECT_LOCATION = "s3:x-amz-website-redirect-location"; + public static final String KEY_X_AMZ_OBJECT_LOCK_MODE = "s3:x-amz-object-lock-mode"; + public static final String KEY_X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE = "s3:x-amz-object-lock-retain-until-date"; + public static final String KEY_X_AMZ_OBJECT_LOCK_REMAINING_RETENTION_DAYS = "s3:x-amz-object-lock-remaining-retention-days"; + public static final String KEY_X_AMZ_OBJECT_LOCK_LEGAL_HOLD = "s3:x-amz-object-lock-legal_hold"; public static final String CHARSET_UTF_8 = "UTF-8"; public static final String CHARSET_UTF_8_LOWER = "utf-8"; @@ -364,6 +395,7 @@ public final class GWConstants { public static final String ISO_8601_TIME_SIMPLE_FORMAT = "yyyy-MM-dd'T'HH:mm:ss'Z'"; public static final String ISO_8601_TIME_FORMAT_MILI = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"; public static final String LIFECYCLE_CONTAIN_TIME = "T00:00:00"; + public static final String LOG_8601_DATE = "8601date : {}"; public static final String HMAC_SHA1 = "HmacSHA1"; public static final String AWS4_HMAC_SHA256 = "AWS4-HMAC-SHA256"; @@ -443,6 +475,7 @@ public final class GWConstants { public static final String X_AMZ_COPY_SOURCE_RANGE = "x-amz-copy-source-range"; public static final String X_AMZ_VERSION_ID = "x-amz-version-id"; public static final String X_AMZ_DELETE_MARKER = "x-amz-delete-marker"; + public static final String X_AMZ_SDK_CHECKSUM_ALGORITHM = "x-amz-sdk-checksum-algorithm"; public static final String USER_METADATA_PREFIX = "x-amz-meta-"; @@ -492,6 +525,7 @@ public final class GWConstants { public static final String KEY_MARKER = "key-marker"; public static final String MAX_UPLOADS = "max-uploads"; public static final String UPLOAD_ID_MARKER = "upload-id-marker"; + public static final String TAG = "tag"; public static final String CONTINUATION_TOKEN = "continuation-token"; public static final String FETCH_OWNER = "fetch-owner"; @@ -609,6 +643,9 @@ public final class GWConstants { public static final String VERSION_CONFIGURATION_XMLNS_ENABLED = "Enabled"; public static final String VERSION_CONFIGURATION_XMLNS_SUSPENDED = "Suspended"; + public static final String TAG_INDEX_CONFIGURATION_XMLNS_DISABLE = "Disabled"; + public static final String TAG_INDEX_CONFIGURATION_XMLNS_ENABLED = "Enabled"; + public static final String TAGGING = "Tagging"; public static final String TAG_SET = "TagSet"; public static final String TAG_SET_ASSIGN = "tagset="; @@ -616,6 +653,7 @@ public final class GWConstants { public static final String XML_TYPE = "type"; public static final String LIST_BUCKET_RESULT = "ListBucketResult"; + public static final String LIST_BUCKET_TAG_SEARCH = "ListBucketTagSearchResult"; public static final String LIST_VERSIONS_RESULT = "ListVersionsResult"; public static final String LIST_ALL_MY_BUCKETS_RESULT = "ListAllMyBucketsResult"; public static final String RETENTION = "Retention"; @@ -631,6 +669,7 @@ public final class GWConstants { public static final String KMS_MASTERKEY_ID = "KMSMasterKeyID"; public static final String BUCKET_KEY_ENABLED = "BucketKeyEnabled"; public static final String OBJECT_LOCK_ENABLED = "ObjectLockEnabled"; + public static final String LEGAL_HOLD = "LegalHold"; public static final String DEFAULT_RETENTION = "DefaultRetention"; public static final String ACCESS_CONTROL_LIST = "AccessControlList"; public static final String DELETE_MARKER_REPLICATION = "DeleteMarkerReplication"; @@ -744,6 +783,7 @@ public final class GWConstants { public static final String LEFT_BRACE = "{"; public static final String RIGHT_BRACE = "}"; public static final String ACCESS_OW = "\"ow\":"; + public static final String ACCESS_OW_EMPTY = "\"ow\":{}"; public static final String ACCESS_ACS = ",\"acs\":"; public static final String ACCESS_ID = "\"id\":"; public static final String ACCESS_COMMA_ID = ",\"id\":"; @@ -842,7 +882,8 @@ public final class GWConstants { public static final int RANGE_LENGTH_INDEX = 1; public static final String UTILITY_EXCHANGE_KEY = "UtilityExchange"; - public static final String MESSAGE_QUEUE_OPTION = "fanout"; + public static final String MESSAGE_QUEUE_OPTION_FANOUT = "fanout"; + public static final String MESSAGE_QUEUE_OPTION_DIRECT = "direct"; // GWConfig public static final String LOG_CONFIG_NOT_EXIST = "Properties file is not exist"; @@ -889,6 +930,7 @@ public final class GWConstants { // CompleteMultipartUpload public static final String LOG_COMPLETE_MULTIPART_UPLOAD_START = "CompleteMultipartUpload ..."; + public static final String LOG_ADMIN_COMPLETE_MULTIPART_UPLOAD_START = "AdmCompleteMultipartUpload ..."; public static final String LOG_COMPLETE_MULTIPART_UPLOAD_PART_NO_EXIST = ": upload part doesn't exist"; public static final String LOG_COMPLETE_MULTIPART_UPLOAD_XML_PARTS_SIZE = "xml parts size : {}"; public static final String LOG_COMPLETE_MULTIPART_UPLOAD_PARTS_SIZE = "parts size : {}"; @@ -920,6 +962,7 @@ public final class GWConstants { // CreateMultipartUpload public static final String LOG_CREATE_MULTIPART_UPLOAD_START = "CreateMultipartUpload ..."; + public static final String LOG_ADMIN_CREATE_MULTIPART_UPLOAD_START = "AdmCreateMultipartUpload ..."; public static final String LOG_CREATE_MULTIPART_UPLOAD_FAILED = "object insert failed(CreateMultipartUpload). bucket={}, object={}"; // DeleteBucket @@ -946,11 +989,15 @@ public final class GWConstants { // DeleteBucketTagging public static final String LOG_DELETE_BUCKET_TAGGING_START = "DeleteBucketTagging ..."; + // DeleteBucketTagIndex + public static final String LOG_DELETE_BUCKET_TAG_INDEX_START = "DeleteBucketTagIndex ..."; + // DeleteBucketWebsite public static final String LOG_DELETE_BUCKET_WEBSITE_START = "DeleteBucketWebsite ..."; // DeleteObject public static final String LOG_DELETE_OBJECT_START = "DeleteObject ..."; + public static final String LOG_ADMIN_DELETE_OBJECT_START = "AdmDeleteObject ..."; public static final String LOG_DELETE_OBJECT = "delete : {}/{}"; public static final String LOG_DELETE_OBJECT_INFO = "versionId : {}, isLastVersion : {}, deleteMarker : {}"; public static final String LOG_DELETE_OBJECT_BUCKET_VERSIONING = "bucket versioning : {}"; @@ -969,12 +1016,14 @@ public final class GWConstants { // DeleteObjectTagging public static final String LOG_DELETE_OBJECT_TAGGING_START = "DeleteObjectTagging ..."; + public static final String LOG_ADMIN_DELETE_OBJECT_TAGGING_START = "AdmDeleteObjectTagging ..."; // DeleteBucketPublicAccessBlock public static final String LOG_DELETE_BUCKET_PUBLIC_ACCESS_BLOCK_START = "DeleteBucketPublicAccessBlock ..."; // GetBucketAcl public static final String LOG_GET_BUCKET_ACL_START = "GetBucketAcl ..."; + public static final String LOG_ADMIN_GET_BUCKET_ACL_START = "AdmGetBucketAcl ..."; // GetBucketCors public static final String LOG_GET_BUCKET_CORS_START = "GetBucketCors ..."; @@ -1009,18 +1058,30 @@ public final class GWConstants { // GetBucketTagging public static final String LOG_GET_BUCKET_TAGGING_START = "GetBucketTagging ..."; + // GetBucketTagIndex + public static final String LOG_GET_BUCKET_TAG_INDEX_START = "GetBucketTagIndex ..."; + // GetBucketVersioning public static final String LOG_GET_BUCKET_VERSIONING_START = "GetBucketVersioning ..."; public static final String LOG_GET_BUCKET_VERSIONING = "bucket({}) versioning : {}"; public static final String LOG_GET_BUCKET_VERSIONING_WRONG = "not defined versioning status: {}"; public static final String LOG_GET_BUCKET_VERSIONING_XML = "xml : {}"; + // GetBucketTagIndexing + public static final String LOG_GET_BUCKET_TAG_INDEX_XML = "xml : {}"; + // GetBucketWebsite public static final String LOG_GET_BUCKET_WEBSITE_START = "GetBucketWebsite ..."; public static final String LOG_GET_BUCKET_WEBSITE = "web : {}"; + // GetBucketLogging + public static final String LOG_GET_BUCKET_LOGGING_START = "GetBucketLogging ..."; + public static final String LOG_GET_BUCKET_LOGGING = "logging : {}"; + public static final String LOG_GET_BUCKET_LOGGING_EMPTY = "\n"; + // GetObjcet public static final String LOG_GET_OBJECT_START = "GetObjcet ..."; + public static final String LOG_ADMIN_GET_OBJECT_START = "AdmGetObjcet ..."; public static final String LOG_GET_OBJECT_CUSTOMER_KEY_NO_MATCH = "encryption-customer-key does not match"; public static final String LOG_GET_OBJECT_IF_MATCH_ETAG = "source etag : {}, IfMatch : {}"; public static final String LOG_GET_OBJECT_IF_NONE_MATCH_ETAG = "source etag : {}, IfNoneMatch : {}"; @@ -1032,12 +1093,17 @@ public final class GWConstants { // GetObjectAcl public static final String LOG_GET_OBJECT_ACL_START = "GetObjectAcl ..."; + public static final String LOG_ADMIN_GET_OBJECT_ACL_START = "AdmGetObjectAcl ..."; // GetObjectRetention public static final String LOG_GET_OBJECT_RETENTION_START = "GetObjectRetention ..."; + // GetObjectLockConfiguration + public static final String LOG_GET_OBJECT_LOCK_CONFIGURATION_START = "GetObjectLockConfiguration ..."; + // GetObjectTagging public static final String LOG_GET_OBJECT_TAGGING_START = "GetObjectTagging ..."; + public static final String LOG_ADMIN_GET_OBJECT_TAGGING_START = "AdmGetObjectTagging ..."; // GetPublicAccessBlock public static final String LOG_GET_PUBLIC_ACCESS_BLOCK_START = "GetPublicAccessBlock ..."; @@ -1047,12 +1113,15 @@ public final class GWConstants { // HeadObject public static final String LOG_HEAD_OBJECT_START = "HeadObject ..."; + public static final String LOG_ADMIN_HEAD_OBJECT_START = "AdmHeadObject ..."; // ListBuckets public static final String LOG_LIST_BUCKETS_START = "ListBuckets ..."; public static final String LOG_LIST_BUCKETS_SIZE = "bucket list size : {}"; public static final String LOG_LIST_BUCKETS_INFO = "{}, {}"; + // ListBucketTag + // ListMultipartUploads public static final String LOG_LIST_MULTIPART_UPLOADS_START = "ListMultipartUploads ..."; public static final String LOG_LIST_MULTIPART_UPLOADS_KEY = "Key : {}"; @@ -1063,6 +1132,8 @@ public final class GWConstants { public static final String LOG_LIST_OBJECT_START = "ListObject ..."; public static final String LOG_LIST_OBJECT_PREFIX_ENCODING = "prefix, encoding type : {}, {}"; + // ListBucketTagSearch + public static final String LOG_LIST_BUCKET_TAG_SEARCH_START = "ListBucketTagSearch ..."; // ListObjectV2 public static final String LOG_LIST_OBJECT_V2_START = "ListObjectV2 ..."; @@ -1071,6 +1142,7 @@ public final class GWConstants { public static final String LOG_LIST_OBJECT_VERSIONS_MAXKEYS = "maxKeys = {}"; public static final String LOG_LIST_OBJECT_VERSIONS_KEY_COUNT = "key count : {}"; public static final String LOG_LIST_OBJECT_VERSIONS_INFO = "object : {}, lastModified : {}, versionId : {}"; + public static final String LOG_LIST_OBJECT_VERSIONS_MARKER = "deleteMarker : {}, lastModified : {}, versionId : {}"; // ListParts public static final String LOG_LIST_PARTS_START = "ListParts ..."; @@ -1097,7 +1169,7 @@ public final class GWConstants { public static final String LOG_PUT_BUCKET_LIFECYCLE_XML = "lifecycle : {}"; // PutBucketObjectLock - public static final String LOG_PUT_BUCKET_OBJECT_LOCK_START = "PutObjectLockConfiguration ..."; + public static final String LOG_PUT_BUCKET_OBJECT_LOCK_START = "PutBucketObjectLock ..."; public static final String LOG_PUT_BUCKET_OBJECT_LOCK = "ObjectLock : {}"; // PutBucketPolicy @@ -1142,14 +1214,22 @@ public final class GWConstants { public static final String LOG_PUT_BUCKET_TAGGING_START = "PutBucketTagging ..."; public static final String LOG_PUT_BUCKET_TAGGING = "tagging xml : {}"; + // PutBucketTagIndex + public static final String LOG_PUT_BUCKET_TAG_INDEX_START = "PutBucketTagIndex ..."; + public static final String LOG_PUT_BUCKET_TAG_INDEX = "tagIndex xml : {}"; + // PutBucketVersioning public static final String LOG_PUT_BUCKET_VERSIONING_START = "PutBucketVersioning ..."; // PutBucketWebsite public static final String LOG_PUT_BUCKET_WEBSITE_START = "PutBucketWebsite ..."; + // PutBucketLogging + public static final String LOG_PUT_BUCKET_LOGGING_START = "PutBucketLogging ..."; + // PutObject public static final String LOG_PUT_OBJECT_START = "PutObject ..."; + public static final String LOG_ADMIN_PUT_OBJECT_START = "AdmPutObject ..."; public static final String LOG_PUT_OBJECT_HASHCODE_ILLEGAL = "HashCode Illegal"; public static final String LOG_PUT_OBJECT_TAGGING_KEY_LENGTH = "key length : {}"; public static final String LOG_PUT_OBJECT_TAGGING_VALUE_LENGTH = "key value length : {}"; @@ -1163,18 +1243,26 @@ public final class GWConstants { // PutObjectAcl public static final String LOG_PUT_OBJECT_ACL_START = "PutObjectAcl ..."; + public static final String LOG_ADMIN_PUT_OBJECT_ACL_START = "AdmPutObjectAcl ..."; + + // PutObjectLegalHold + // PutObjectRetention + public static final String LOG_PUT_OBJECT_LEGALHOLD_START = "PutObjectLegalHold ..."; // PutObjectRetention public static final String LOG_PUT_OBJECT_RETENTION_START = "PutObjectRetention ..."; + public static final String LOG_CUR_NEW_DATE = "cur : {}, new : {}"; // PutObjectTagging public static final String LOG_PUT_OBJECT_TAGGING_START = "PutObjectTagging ..."; + public static final String LOG_ADMIN_PUT_OBJECT_TAGGING_START = "AdmPutObjectTagging ..."; // PutBucketPublicAccessBlock public static final String LOG_PUT_BUCKET_PUBLIC_ACCESS_BLOCK_START = "PutBucketPublicAccessBlock ..."; // UploadPart public static final String LOG_UPLOAD_PART_START = "UploadPart ..."; + public static final String LOG_ADMIN_UPLOAD_PART_START = "AdmUploadPart ..."; public static final String LOG_UPLOAD_PART_WRONG_PART_NUMBER = " : Part number must be an integer between 1 and 10000, inclusive"; public static final int MAX_PARTS_SIZE = 10000; public static final String ARGMENT_NAME = "ArgumentName"; @@ -1187,6 +1275,9 @@ public final class GWConstants { public static final String LOG_UPLOAD_PART_COPY_SOURCE = "copySource : {}"; public static final String LOG_UPLOAD_PART_COPY_SOURCE_RANGE = "copy source range : {}, file size : {}"; + // RestoreObject + public static final String LOG_RESTORE_OBJECT_START = "RestoreObject ..."; + // PostObject public static final String LOG_POST_OBJECT_START = "PostObject ..."; @@ -1201,6 +1292,7 @@ public final class GWConstants { public static final String LOG_UTILS_TIME_SKEWED = "time skewed : {}, now : {}"; public static final String LOG_UTILS_UNDEFINED_DB = "undefined db repository."; public static final String LOG_UTILS_KEY = "key : {}"; + public static final String LOG_UTILS_HAS_KEYWORD_ACL = "has keyword : {}"; public static final String LOG_UTILS_SOURCE_ACL = "source acl : {}"; public static final String LOG_UTILS_CANNED_ACL = "cannedAcl : {}"; public static final String LOG_UTILS_ACL_XML = "aclXml : {}"; @@ -1232,6 +1324,7 @@ public final class GWConstants { public static final String LOG_DATA_PART_NUMBER_MARKER_NULL = "partNumberMarker is null or empty"; public static final String LOG_DATA_LIFECYCLE_R1_STATUS = "rl.status : {}"; public static final String LOG_DATA_LIFECYCLE_LCC_RULE_SIZE = "lcc.rules.size : {}, id.size : {}"; + public static final String LOG_DATA_TAG_NULL = "tag is null or empty"; // ObjManagerHelper public static final String LOG_OBJMANAGER_COUNT = "objManager count : {}"; @@ -1262,11 +1355,11 @@ public final class GWConstants { public static final String LOG_OSDCLIENT_MANAGER_OSD_SERVER_IP = "add osd server ip : {}"; // S3ObjectOperation - public static final String FILE_ATTRIBUTE_REPLICATION = "replication"; - public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID = "replica-diskid"; - public static final String FILE_ATTRUBUTE_REPLICATION_PRIMARY = "primary"; - public static final String FILE_ATTRIBUTE_REPLICATION_REPLICA = "replica"; - public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL = "null"; + // public static final String FILE_ATTRIBUTE_REPLICATION = "replication"; + // public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID = "replica-diskid"; + // public static final String FILE_ATTRUBUTE_REPLICATION_PRIMARY = "primary"; + // public static final String FILE_ATTRIBUTE_REPLICATION_REPLICA = "replica"; + // public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL = "null"; public static final String LOG_S3OBJECT_OPERATION_FILE_SIZE = "get obeject file size : {}"; public static final String LOG_S3OBJECT_OPERATION_RANGE = "offset : {}, length : {}"; public static final String LOG_S3OBJECT_OPERATION_OBJECT_PRIMARY_INFO = "obj primary : {}"; @@ -1287,11 +1380,6 @@ public final class GWConstants { public static final String LOG_S3OBJECT_OPERATION_COPY_SOURCE_RANGE = "copySourceRange : {}"; public static final String LOG_S3OBJECT_OPERATION_DISK_IP_NULL = "diskid : {} -> ip is null. check disk pool"; public static final String LOG_S3OBJECT_OPERATION_DISK_PATH_NULL = "diskid : {} -> path is null. check disk pool"; - public static final String ZUNFEC = "zunfec -o "; - public static final String ZFEC_0 = ".0_4.fec"; - public static final String ZFEC_1 = ".1_4.fec"; - public static final String ZFEC_2 = ".2_4.fec"; - public static final String ZFEC_3 = ".3_4.fec"; public static final String LOG_S3OBJECT_OPERATION_ZUNFEC_COMMAND = "command : {}"; public static final String LOG_S3OBJECT_OPERATION_ZUNFEC_DECODE = "DECODE EC : {}"; public static final String LOG_S3OBJECT_OPERATION_ZUNFEC_DECODE_EXIT_VALUE = "DECODE exit : {}"; @@ -1379,4 +1467,35 @@ public final class GWConstants { public static final String GWPORTAL_RECEIVED_USER_REMOVED = "removed"; public static final String LOG_GWPORTAL_RECEIVED_USER_WRONG_ROUTING_KEY = "wrong routingKey : {}"; public static final String LOG_GWPORTAL_RECEIVED_USER_DATA = "Id:{}, Name:{}, Email:{}, AccessKey:{}, SecretKey:{}"; + + // log + public static final String GW_LOG_USER_NAME = "UserName"; + public static final String GW_LOG_BUCKET_NAME = "BucketName"; + public static final String GW_LOG_DATE = "Date"; + public static final String GW_LOG_REMOTE_HOST = "RemoteHost"; + public static final String GW_LOG_REQUEST_USER = "RequestUser"; + public static final String GW_LOG_REQUEST_ID = "RequestId"; + public static final String GW_LOG_OPERATION = "Operation"; + public static final String GW_LOG_OBJECT_NAME = "ObjectName"; + public static final String GW_LOG_REQUEST_URI = "RequestURI"; + public static final String GW_LOG_STATUS_CODE = "StatusCode"; + public static final String GW_LOG_ERROR_CODE = "ErrorCode"; + public static final String GW_LOG_RESPONSE_LENGTH = "ResponseLength"; + public static final String GW_LOG_OBJECT_LENGTH = "ObjectLength"; + public static final String GW_LOG_TOTAL_TIME = "TotalTime"; + public static final String GW_LOG_REQUEST_LENGTH = "RequestLength"; + public static final String GW_LOG_REFERER = "Referer"; + public static final String GW_LOG_USER_AGENT = "UserAgent"; + public static final String GW_LOG_VERSION_ID = "VersionId"; + public static final String GW_LOG_HOST_ID = "HostId"; + public static final String GW_LOG_SIGN = "Sign"; + public static final String GW_LOG_SSL_GROUP = "SSLGroup"; + public static final String GW_LOG_SIGN_TYPE = "SignType"; + public static final String GW_LOG_END_POINT = "EndPoint"; + public static final String GW_LOG_TLS_VERSION = "TLSVersion"; + + public static final String LOG_KEY = "key : {}"; + public static final String LOG_VALUE = "value : {}"; + public static final String LOG_T_KEY = "t.key : {}"; + public static final String LOG_T_VALUE = "t.value : {}"; } diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/GWLogging.java b/core/src/com/pspace/ifs/ksan/gw/utils/GWLogging.java new file mode 100644 index 00000000..98cca6ee --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/gw/utils/GWLogging.java @@ -0,0 +1,215 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.gw.utils; + +import java.util.Date; + +import com.pspace.ifs.ksan.libs.config.AgentConfig; +import com.pspace.ifs.ksan.libs.PrintStack; +import com.pspace.ifs.ksan.libs.mq.MQSender; + +import com.pspace.ifs.ksan.gw.mq.MessageQueueSender; +import org.json.simple.JSONObject; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.pspace.ifs.ksan.gw.identity.S3Parameter; +import com.google.common.base.Strings; + +public class GWLogging { + private static final Logger logger = LoggerFactory.getLogger(GWLogging.class); + private MessageQueueSender MQS; + // private static MQSender mqSender; + + public static GWLogging getInstance() { + return LazyHolder.INSTANCE; + } + + private static class LazyHolder { + private static final GWLogging INSTANCE = new GWLogging(); + } + + private GWLogging() { + try { + // mqSender = new MQSender(AgentConfig.getInstance().getMQHost(), + // Integer.parseInt(AgentConfig.getInstance().getMQPort()), + // AgentConfig.getInstance().getMQUser(), + // AgentConfig.getInstance().getMQPassword(), + // GWConstants.MQUEUE_LOG_EXCHANGE_NAME, + // GWConstants.MESSAGE_QUEUE_OPTION_DIRECT, + // GWConstants.MQUEUE_NAME_GW_LOG_ADD); + MQS = new MessageQueueSender(AgentConfig.getInstance().getMQHost(), + Integer.parseInt(AgentConfig.getInstance().getMQPort()), + AgentConfig.getInstance().getMQUser(), + AgentConfig.getInstance().getMQPassword(), + GWConstants.MQUEUE_LOG_EXCHANGE_NAME, + GWConstants.MESSAGE_QUEUE_OPTION_DIRECT, + GWConstants.MQUEUE_NAME_GW_LOG_ADD); + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } + + public void sendLog(S3Parameter s3Parameter) { + JSONObject object = new JSONObject(); + + // bucket owner name + if (s3Parameter.getBucket() != null && !Strings.isNullOrEmpty(s3Parameter.getBucket().getUserName())) { + object.put(GWConstants.GW_LOG_USER_NAME, s3Parameter.getBucket().getUserName()); + } else { + object.put(GWConstants.GW_LOG_USER_NAME, GWConstants.EMPTY_STRING); + } + + // bucket name + if (s3Parameter.getBucket() != null && !Strings.isNullOrEmpty(s3Parameter.getBucket().getBucket())) { + object.put(GWConstants.GW_LOG_BUCKET_NAME, s3Parameter.getBucket().getBucket()); + } else { + object.put(GWConstants.GW_LOG_BUCKET_NAME, GWConstants.EMPTY_STRING); + } + + // date + Date date = new Date(); + object.put(GWConstants.GW_LOG_DATE, date.toString()); + + // remote host + if (!Strings.isNullOrEmpty(s3Parameter.getRemoteHost())) { + object.put(GWConstants.GW_LOG_REMOTE_HOST, s3Parameter.getRemoteHost()); + } else { + object.put(GWConstants.GW_LOG_REMOTE_HOST, GWConstants.EMPTY_STRING); + } + + // request user + if (s3Parameter.getUser() != null && !Strings.isNullOrEmpty(s3Parameter.getUser().getUserName())) { + object.put(GWConstants.GW_LOG_REQUEST_USER, s3Parameter.getUser().getUserName()); + } else { + object.put(GWConstants.GW_LOG_REQUEST_USER, GWConstants.EMPTY_STRING); + } + + // request id + if (!Strings.isNullOrEmpty(s3Parameter.getRequestID())) { + object.put(GWConstants.GW_LOG_REQUEST_ID, String.valueOf(s3Parameter.getRequestID())); + } else { + object.put(GWConstants.GW_LOG_REQUEST_ID, GWConstants.EMPTY_STRING); + } + + // operation + if (!Strings.isNullOrEmpty(s3Parameter.getOperation())) { + object.put(GWConstants.GW_LOG_OPERATION, s3Parameter.getOperation()); + } else { + object.put(GWConstants.GW_LOG_OPERATION, GWConstants.EMPTY_STRING); + } + + // object name + if (!Strings.isNullOrEmpty(s3Parameter.getObjectName())) { + object.put(GWConstants.GW_LOG_OBJECT_NAME, s3Parameter.getObjectName()); + } else { + object.put(GWConstants.GW_LOG_OBJECT_NAME, GWConstants.EMPTY_STRING); + } + + // request uri + if (!Strings.isNullOrEmpty(s3Parameter.getRequestURI())) { + object.put(GWConstants.GW_LOG_REQUEST_URI, s3Parameter.getRequestURI()); + } else { + object.put(GWConstants.GW_LOG_REQUEST_URI, GWConstants.EMPTY_STRING); + } + + // reponse status code + object.put(GWConstants.GW_LOG_STATUS_CODE, s3Parameter.getStatusCode()); + + // response error code + if (!Strings.isNullOrEmpty(s3Parameter.getErrorCode())) { + object.put(GWConstants.GW_LOG_ERROR_CODE, s3Parameter.getErrorCode()); + } else { + object.put(GWConstants.GW_LOG_ERROR_CODE, GWConstants.EMPTY_STRING); + } + + // response length + object.put(GWConstants.GW_LOG_RESPONSE_LENGTH, s3Parameter.getResponseSize()); + + // object length + if (s3Parameter.getFileSize() > 0) { + object.put(GWConstants.GW_LOG_OBJECT_LENGTH, s3Parameter.getFileSize()); + } else { + object.put(GWConstants.GW_LOG_OBJECT_LENGTH, 0L); + } + + // total time + object.put(GWConstants.GW_LOG_TOTAL_TIME, System.currentTimeMillis() - s3Parameter.getStartTime()); + + // request length + object.put(GWConstants.GW_LOG_REQUEST_LENGTH, s3Parameter.getRequestSize()); + + // referer + if (!Strings.isNullOrEmpty(s3Parameter.getReferer())) { + object.put(GWConstants.GW_LOG_REFERER, s3Parameter.getReferer()); + } else { + object.put(GWConstants.GW_LOG_REFERER, GWConstants.EMPTY_STRING); + } + + // User Agent + if (!Strings.isNullOrEmpty(s3Parameter.getUserAgent())) { + object.put(GWConstants.GW_LOG_USER_AGENT, s3Parameter.getUserAgent()); + } else { + object.put(GWConstants.GW_LOG_USER_AGENT, GWConstants.EMPTY_STRING); + } + + // Version id + if (!Strings.isNullOrEmpty(s3Parameter.getVersionId())) { + object.put(GWConstants.GW_LOG_VERSION_ID, s3Parameter.getVersionId()); + } else { + object.put(GWConstants.GW_LOG_VERSION_ID, GWConstants.EMPTY_STRING); + } + + // Host ID + if (!Strings.isNullOrEmpty(s3Parameter.getHostID())) { + object.put(GWConstants.GW_LOG_HOST_ID, s3Parameter.getHostID()); + } else { + object.put(GWConstants.GW_LOG_HOST_ID, GWConstants.EMPTY_STRING); + } + + // Sign Version + if (!Strings.isNullOrEmpty(s3Parameter.getSignVersion())) { + object.put(GWConstants.GW_LOG_SIGN, s3Parameter.getSignVersion()); + } else { + object.put(GWConstants.GW_LOG_SIGN, GWConstants.EMPTY_STRING); + } + + // ssl_group + object.put(GWConstants.GW_LOG_SSL_GROUP, GWConstants.EMPTY_STRING); + + // sign type + if (!Strings.isNullOrEmpty(s3Parameter.getAuthorization())) { + object.put(GWConstants.GW_LOG_SIGN_TYPE, GWConstants.AUTH_HEADER); + } else if (!Strings.isNullOrEmpty(s3Parameter.getxAmzAlgorithm())) { + object.put(GWConstants.GW_LOG_SIGN_TYPE, GWConstants.QUERY_STRING); + } else { + object.put(GWConstants.GW_LOG_SIGN_TYPE, GWConstants.EMPTY_STRING); + } + + // endpoint + if (!Strings.isNullOrEmpty(s3Parameter.getHostName())) { + object.put(GWConstants.GW_LOG_END_POINT, s3Parameter.getHostName()); + } else { + object.put(GWConstants.GW_LOG_END_POINT, GWConstants.EMPTY_STRING); + } + + // tls version + object.put(GWConstants.GW_LOG_TLS_VERSION, GWConstants.EMPTY_STRING); + + logger.debug("log - {}", object.toString()); + try { + MQS.send(object.toString(), GWConstants.MQUEUE_NAME_GW_LOG_ADD); + // mqSender.send(object.toString(), GWConstants.MQUEUE_NAME_GW_LOG_ADD); + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } +} diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/GWPortal.java b/core/src/com/pspace/ifs/ksan/gw/utils/GWPortal.java index 283104da..8f96cbbd 100644 --- a/core/src/com/pspace/ifs/ksan/gw/utils/GWPortal.java +++ b/core/src/com/pspace/ifs/ksan/gw/utils/GWPortal.java @@ -31,6 +31,7 @@ import com.pspace.ifs.ksan.libs.disk.DiskPool; import com.pspace.ifs.ksan.libs.disk.Server; import com.pspace.ifs.ksan.libs.HeartbeatManager; +import com.pspace.ifs.ksan.libs.Constants; import com.google.common.base.Strings; import org.apache.http.HttpResponse; @@ -553,10 +554,24 @@ public void getDiskPoolsDetails() { for (int i = 0; i < jsonItems.size(); i++) { JSONObject item = (JSONObject)jsonItems.get(i); - DiskPool diskPool = new DiskPool((String)item.get(DiskPool.ID), - (String)item.get(DiskPool.NAME), - (String)item.get(DiskPool.DISK_POOL_TYPE), - (String)item.get(DiskPool.REPLICATION_TYPE)); + JSONObject jsonEC = (JSONObject)item.get(DiskPool.EC); + + DiskPool diskPool = null; + if (jsonEC != null) { + logger.info("jsonEC : {}", jsonEC.toString()); + diskPool = new DiskPool((String)item.get(DiskPool.ID), + (String)item.get(DiskPool.NAME), + (String)item.get(DiskPool.DISK_POOL_TYPE), + (String)item.get(DiskPool.REPLICATION_TYPE), + (int)(long)jsonEC.get(DiskPool.EC_M), + (int)(long)jsonEC.get(DiskPool.EC_K)); + } else { + diskPool = new DiskPool((String)item.get(DiskPool.ID), + (String)item.get(DiskPool.NAME), + (String)item.get(DiskPool.DISK_POOL_TYPE), + (String)item.get(DiskPool.REPLICATION_TYPE)); + } + JSONArray jsonServers = (JSONArray)item.get(DiskPool.SERVERS); if (jsonServers != null && jsonServers.size() == 0) { logger.info("diskpools -- servers is empty"); @@ -589,11 +604,11 @@ public void getDiskPoolsDetails() { for (Server server : diskpool.getServerList()) { if (GWUtils.getLocalIP().equals(server.getIp())) { for (Disk disk : server.getDiskList()) { - File file = new File(disk.getPath() + GWConstants.SLASH + GWConstants.OBJ_DIR); + File file = new File(disk.getPath() + GWConstants.SLASH + Constants.OBJ_DIR); file.mkdirs(); - file = new File(disk.getPath() + GWConstants.SLASH + GWConstants.TEMP_DIR); + file = new File(disk.getPath() + GWConstants.SLASH + Constants.TEMP_DIR); file.mkdirs(); - file = new File(disk.getPath() + GWConstants.SLASH + GWConstants.TRASH_DIR); + file = new File(disk.getPath() + GWConstants.SLASH + Constants.TRASH_DIR); file.mkdirs(); } } @@ -654,41 +669,6 @@ public void getS3Users() { } } - // public S3User getS3User(String id) { - // try { - // HttpClient client = HttpClients - // .custom() - // .setSSLContext(new SSLContextBuilder().loadTrustMaterial(null, TrustAllStrategy.INSTANCE).build()) - // .setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE) - // .build(); - - // HttpGet getRequest = new HttpGet(GWConstants.HTTPS + agentConfig.getPortalIp() + GWConstants.COLON + agentConfig.getPortalPort() + GWConstants.PORTAL_REST_API_KSAN_USERS + GWConstants.SLASH + id); - // getRequest.addHeader(GWConstants.AUTHORIZATION, agentConfig.getPortalKey()); - - // HttpResponse response = client.execute(getRequest); - // if (response.getStatusLine().getStatusCode() == 200) { - // ResponseHandler handler = new BasicResponseHandler(); - // String body = handler.handleResponse(response); - // JSONParser parser = new JSONParser(); - // JSONObject jsonObject = (JSONObject)parser.parse(body); - // JSONObject jsonData = (JSONObject)jsonObject.get(S3User.DATA); - // JSONArray jsonUserDiskpools = (JSONArray)jsonData.get(S3User.USER_DISK_POOLS); - // S3User user = new S3User((String)jsonData.get(S3User.USER_ID), - // (String)jsonData.get(S3User.USER_NAME), - // (String)jsonData.get(S3User.USER_EMAIL), - // (String)jsonData.get(S3User.ACCESS_KEY), - // (String)jsonData.get(S3User.ACCESS_SECRET), - // jsonUserDiskpools); - // logger.info(GWConstants.LOG_GWPORTAL_RECEIVED_USER_DATA, user.getUserId(), user.getUserName(), user.getUserEmail(), user.getAccessKey(), user.getAccessSecret()); - // return user; - // } - // throw new RuntimeException(new RuntimeException()); - // } catch (Exception e) { - // PrintStack.logging(logger, e); - // throw new RuntimeException(e); - // } - // } - public void getS3Regions() { try { HttpClient client = HttpClients diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/GWUtils.java b/core/src/com/pspace/ifs/ksan/gw/utils/GWUtils.java index 9e7092d5..5e18e216 100644 --- a/core/src/com/pspace/ifs/ksan/gw/utils/GWUtils.java +++ b/core/src/com/pspace/ifs/ksan/gw/utils/GWUtils.java @@ -53,6 +53,7 @@ import com.pspace.ifs.ksan.gw.exception.GWErrorCode; import com.pspace.ifs.ksan.gw.exception.GWException; import com.pspace.ifs.ksan.gw.format.AccessControlPolicy; +import com.pspace.ifs.ksan.gw.format.AccessControlPolicy.Owner; import com.pspace.ifs.ksan.gw.format.AccessControlPolicyJson; import com.pspace.ifs.ksan.gw.format.CORSConfiguration; import com.pspace.ifs.ksan.gw.format.Policy; @@ -69,6 +70,7 @@ import com.pspace.ifs.ksan.libs.disk.Disk; import com.pspace.ifs.ksan.libs.disk.DiskPool; import com.pspace.ifs.ksan.libs.disk.Server; +import com.pspace.ifs.ksan.libs.Constants; public class GWUtils { @@ -91,89 +93,89 @@ private static void addResponseHeaderWithOverride( public static void addMetadataToResponse(HttpServletRequest request, HttpServletResponse response, S3Metadata s3Metadata, List ContentLength_Headers, Long streamsize) { - addResponseHeaderWithOverride(request, response, - HttpHeaders.CACHE_CONTROL, GWConstants.RESPONSE_CACHE_CONTROL, - s3Metadata.getCacheControl()); - addResponseHeaderWithOverride(request, response, - HttpHeaders.CONTENT_ENCODING, GWConstants.RESPONSE_CONTENT_ENCODING, - s3Metadata.getContentEncoding()); - addResponseHeaderWithOverride(request, response, - HttpHeaders.CONTENT_LANGUAGE, GWConstants.RESPONSE_CONTENT_LANGUAGE, - s3Metadata.getContentLanguage()); - addResponseHeaderWithOverride(request, response, - HttpHeaders.CONTENT_DISPOSITION, GWConstants.RESPONSE_CONTENT_DISPOSITION, - s3Metadata.getContentDisposition()); + // addResponseHeaderWithOverride(request, response, + // HttpHeaders.CACHE_CONTROL, GWConstants.RESPONSE_CACHE_CONTROL, + // s3Metadata.getCacheControl()); + // addResponseHeaderWithOverride(request, response, + // HttpHeaders.CONTENT_ENCODING, GWConstants.RESPONSE_CONTENT_ENCODING, + // s3Metadata.getContentEncoding()); + // addResponseHeaderWithOverride(request, response, + // HttpHeaders.CONTENT_LANGUAGE, GWConstants.RESPONSE_CONTENT_LANGUAGE, + // s3Metadata.getContentLanguage()); + // addResponseHeaderWithOverride(request, response, + // HttpHeaders.CONTENT_DISPOSITION, GWConstants.RESPONSE_CONTENT_DISPOSITION, + // s3Metadata.getContentDisposition()); - // TODO: handles only a single range due to jclouds limitations - Collection contentRanges = ContentLength_Headers; - if (ContentLength_Headers != null && !contentRanges.isEmpty()) { - for (String contents : ContentLength_Headers) { - response.addHeader(HttpHeaders.CONTENT_RANGE, contents); - } + // // TODO: handles only a single range due to jclouds limitations + // Collection contentRanges = ContentLength_Headers; + // if (ContentLength_Headers != null && !contentRanges.isEmpty()) { + // for (String contents : ContentLength_Headers) { + // response.addHeader(HttpHeaders.CONTENT_RANGE, contents); + // } - response.addHeader(HttpHeaders.ACCEPT_RANGES, GWConstants.BYTES); - response.addHeader(HttpHeaders.CONTENT_LENGTH, streamsize.toString()); - } else { - response.addHeader(HttpHeaders.CONTENT_LENGTH, s3Metadata.getContentLength().toString()); - } + // response.addHeader(HttpHeaders.ACCEPT_RANGES, GWConstants.BYTES); + // response.addHeader(HttpHeaders.CONTENT_LENGTH, streamsize.toString()); + // } else { + // response.addHeader(HttpHeaders.CONTENT_LENGTH, s3Metadata.getContentLength().toString()); + // } - String overrideContentType = request.getParameter(GWConstants.RESPONSE_CONTENT_TYPE); - response.setContentType(overrideContentType != null ? overrideContentType : s3Metadata.getContentType()); + // String overrideContentType = request.getParameter(GWConstants.RESPONSE_CONTENT_TYPE); + // response.setContentType(overrideContentType != null ? overrideContentType : s3Metadata.getContentType()); - if (s3Metadata.getCustomerAlgorithm() != null ) { - response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, s3Metadata.getCustomerAlgorithm()); - } + // if (s3Metadata.getCustomerAlgorithm() != null ) { + // response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, s3Metadata.getCustomerAlgorithm()); + // } - if (s3Metadata.getCustomerKey() != null ) { - response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, s3Metadata.getCustomerKey()); - } + // if (s3Metadata.getCustomerKey() != null ) { + // response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, s3Metadata.getCustomerKey()); + // } - if (s3Metadata.getCustomerKeyMD5() != null ) { - response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, s3Metadata.getCustomerKeyMD5()); - } + // if (s3Metadata.getCustomerKeyMD5() != null ) { + // response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, s3Metadata.getCustomerKeyMD5()); + // } - if (s3Metadata.getServersideEncryption() != null ) { - response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION, s3Metadata.getServersideEncryption()); - } + // if (s3Metadata.getServersideEncryption() != null ) { + // response.addHeader(GWConstants.X_AMZ_SERVER_SIDE_ENCRYPTION, s3Metadata.getServersideEncryption()); + // } - if (s3Metadata.getLockMode() != null) { - response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_MODE, s3Metadata.getLockMode()); - } + // if (s3Metadata.getLockMode() != null) { + // response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_MODE, s3Metadata.getLockMode()); + // } - if (s3Metadata.getLockExpires() != null) { - response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, s3Metadata.getLockExpires()); - } + // if (s3Metadata.getLockExpires() != null) { + // response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, s3Metadata.getLockExpires()); + // } - if (s3Metadata.getLegalHold() != null) { - response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_LEGAL_HOLD, s3Metadata.getLegalHold()); - } + // if (s3Metadata.getLegalHold() != null) { + // response.addHeader(GWConstants.X_AMZ_OBJECT_LOCK_LEGAL_HOLD, s3Metadata.getLegalHold()); + // } - if (s3Metadata.getUserMetadataMap() != null ) { - for (Map.Entry entry : s3Metadata.getUserMetadataMap().entrySet()) { - response.addHeader(entry.getKey(), entry.getValue()); - logger.info(GWConstants.LOG_UTILS_USER_META_DATA, entry.getKey(), entry.getValue()); - } - } + // if (s3Metadata.getUserMetadataMap() != null ) { + // for (Map.Entry entry : s3Metadata.getUserMetadataMap().entrySet()) { + // response.addHeader(entry.getKey(), entry.getValue()); + // logger.info(GWConstants.LOG_UTILS_USER_META_DATA, entry.getKey(), entry.getValue()); + // } + // } response.addHeader(HttpHeaders.ETAG, maybeQuoteETag(s3Metadata.getETag())); - String overrideExpires = request.getParameter(GWConstants.RESPONSE_EXPIRES); - if (overrideExpires != null) { - response.addHeader(HttpHeaders.EXPIRES, overrideExpires); - } else { - Date expires = s3Metadata.getExpires(); - if (expires != null) { - response.addDateHeader(HttpHeaders.EXPIRES, expires.getTime()); - } - } + // String overrideExpires = request.getParameter(GWConstants.RESPONSE_EXPIRES); + // if (overrideExpires != null) { + // response.addHeader(HttpHeaders.EXPIRES, overrideExpires); + // } else { + // Date expires = s3Metadata.getExpires(); + // if (expires != null) { + // response.addDateHeader(HttpHeaders.EXPIRES, expires.getTime()); + // } + // } - response.addDateHeader(HttpHeaders.LAST_MODIFIED, s3Metadata.getLastModified().getTime()); + // response.addDateHeader(HttpHeaders.LAST_MODIFIED, s3Metadata.getLastModified().getTime()); - if (s3Metadata.getTaggingCount() != null) { - response.addHeader(GWConstants.X_AMZ_TAGGING_COUNT, s3Metadata.getTaggingCount()); - } + // if (s3Metadata.getTaggingCount() != null) { + // response.addHeader(GWConstants.X_AMZ_TAGGING_COUNT, s3Metadata.getTaggingCount()); + // } - response.addHeader(GWConstants.X_AMZ_VERSION_ID, s3Metadata.getVersionId()); + // response.addHeader(GWConstants.X_AMZ_VERSION_ID, s3Metadata.getVersionId()); } /** Parse ISO 8601 timestamp into seconds since 1970. */ @@ -201,6 +203,31 @@ public static long parseIso8601(String date, S3Parameter s3Parameter) throws GWE } } + public static long parseTime8601(String date) { + SimpleDateFormat formatter = null; + if (date == null) { + return 0; + } + + if (date.length() >= 23) { + formatter = new SimpleDateFormat(GWConstants.ISO_8601_TIME_FORMAT_MILI); + } else if (date.contains(":") && date.length() < 23) { + formatter = new SimpleDateFormat(GWConstants.ISO_8601_TIME_SIMPLE_FORMAT); + } else if (!date.contains(":") && date.length() < 23) { + formatter = new SimpleDateFormat(GWConstants.ISO_8601_TIME_FORMAT); + } + + formatter.setTimeZone(TimeZone.getTimeZone(GWConstants.UTC)); + logger.debug(GWConstants.LOG_8601_DATE, date); + try { + return formatter.parse(date).getTime() / 1000; + } catch (ParseException pe) { + PrintStack.logging(logger, pe); + } + + return 0; + } + public static void isTimeSkewed(long date, int maxTimeSkew, S3Parameter s3Parameter) throws GWException { if (date < 0) { throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); @@ -234,8 +261,7 @@ public static boolean startsField(String string, String field) { } public static boolean constantTimeEquals(String x, String y) { - return MessageDigest.isEqual(x.getBytes(StandardCharsets.UTF_8), - y.getBytes(StandardCharsets.UTF_8)); + return MessageDigest.isEqual(x.getBytes(StandardCharsets.UTF_8), y.getBytes(StandardCharsets.UTF_8)); } // Encode blob name if client requests it. This allows for characters @@ -256,14 +282,14 @@ public static String encodeObjectName(String encodingType, String blobName) { } } - public static GWDB getDBInstance() { - if (GWConfig.getInstance().getDbRepository().equalsIgnoreCase(GWConstants.MARIADB)) { - return MariaDB.getInstance(); - } else { - logger.error(GWConstants.LOG_UTILS_UNDEFINED_DB); - return null; - } - } + // public static GWDB getDBInstance() { + // if (GWConfig.getInstance().getDbRepository().equalsIgnoreCase(GWConstants.MARIADB)) { + // return MariaDB.getInstance(); + // } else { + // logger.error(GWConstants.LOG_UTILS_UNDEFINED_DB); + // return null; + // } + // } public static boolean likematch(String first, String second) { // If we reach at the end of both strings, @@ -737,6 +763,263 @@ protected static void readAclHeader(String grantstr, String permission, AccessCo } public static String makeAclXml(AccessControlPolicy accessControlPolicy, + AccessControlPolicy preAccessControlPolicy, + boolean hasKeyWord, + String getAclXml, + String cannedAcl, + Bucket bucketInfo, + String userId, + String userName, + String getGrantRead, + String getGrantWrite, + String getGrantFullControl, + String getGrantReadAcp, + String getGrantWriteAcp, + S3Parameter s3Parameter, + boolean isAcl) throws GWException { + + PublicAccessBlockConfiguration pabc = null; + if (bucketInfo != null && !Strings.isNullOrEmpty(bucketInfo.getAccess())) { + try { + pabc = new XmlMapper().readValue(bucketInfo.getAccess(), PublicAccessBlockConfiguration.class); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } + + logger.info(GWConstants.LOG_UTILS_HAS_KEYWORD_ACL, hasKeyWord); + logger.info(GWConstants.LOG_UTILS_CANNED_ACL, cannedAcl); + logger.info(GWConstants.LOG_UTILS_ACL_XML, getAclXml); + logger.info("isAcl : {}", isAcl); + + if (preAccessControlPolicy != null && preAccessControlPolicy.owner != null) { + accessControlPolicy.owner.id = preAccessControlPolicy.owner.id; + accessControlPolicy.owner.displayName = preAccessControlPolicy.owner.displayName; + } else { + accessControlPolicy.owner.id = userId; + accessControlPolicy.owner.displayName = userName; + } + + String aclXml = null; + if (!hasKeyWord) { + aclXml = getAclXml; + } + if (Strings.isNullOrEmpty(cannedAcl)) { + if (Strings.isNullOrEmpty(aclXml)) { + if (Strings.isNullOrEmpty(getGrantRead) + && Strings.isNullOrEmpty(getGrantWrite) + && Strings.isNullOrEmpty(getGrantReadAcp) + && Strings.isNullOrEmpty(getGrantWriteAcp) + && Strings.isNullOrEmpty(getGrantFullControl)) { + Grant priUser = new Grant(); + priUser.grantee = new Grantee(); + priUser.grantee.type = GWConstants.CANONICAL_USER; + priUser.grantee.id = accessControlPolicy.owner.id; + priUser.grantee.displayName = accessControlPolicy.owner.displayName; + priUser.permission = GWConstants.GRANT_FULL_CONTROL; + accessControlPolicy.aclList.grants.add(priUser); + } + } + } else { + if (GWConstants.CANNED_ACLS_PRIVATE.equalsIgnoreCase(cannedAcl)) { + Grant priUser = new Grant(); + priUser.grantee = new Grantee(); + priUser.grantee.type = GWConstants.CANONICAL_USER; + priUser.grantee.id = accessControlPolicy.owner.id; + priUser.grantee.displayName = accessControlPolicy.owner.displayName; + priUser.permission = GWConstants.GRANT_FULL_CONTROL; + accessControlPolicy.aclList.grants.add(priUser); + } else if (GWConstants.CANNED_ACLS_PUBLIC_READ.equalsIgnoreCase(cannedAcl)) { + if (pabc != null && GWConstants.STRING_TRUE.equalsIgnoreCase(pabc.BlockPublicAcls)) { + logger.info(GWConstants.LOG_ACCESS_DENIED_PUBLIC_ACLS); + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + Grant priUser = new Grant(); + priUser.grantee = new Grantee(); + priUser.grantee.type = GWConstants.CANONICAL_USER; + priUser.grantee.id = accessControlPolicy.owner.id; + priUser.grantee.displayName = accessControlPolicy.owner.displayName; + priUser.permission = GWConstants.GRANT_FULL_CONTROL; + accessControlPolicy.aclList.grants.add(priUser); + + Grant pubReadUser = new Grant(); + pubReadUser.grantee = new Grantee(); + pubReadUser.grantee.type = GWConstants.GROUP; + pubReadUser.grantee.uri = GWConstants.AWS_GRANT_URI_ALL_USERS; + pubReadUser.permission = GWConstants.GRANT_READ; + accessControlPolicy.aclList.grants.add(pubReadUser); + } else if (GWConstants.CANNED_ACLS_PUBLIC_READ_WRITE.equalsIgnoreCase(cannedAcl)) { + if (pabc != null && GWConstants.STRING_TRUE.equalsIgnoreCase(pabc.BlockPublicAcls)) { + logger.info(GWConstants.LOG_ACCESS_DENIED_PUBLIC_ACLS); + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + Grant priUser = new Grant(); + priUser.grantee = new Grantee(); + priUser.grantee.type = GWConstants.CANONICAL_USER; + priUser.grantee.id = accessControlPolicy.owner.id; + priUser.grantee.displayName = accessControlPolicy.owner.displayName; + priUser.permission = GWConstants.GRANT_FULL_CONTROL; + accessControlPolicy.aclList.grants.add(priUser); + + Grant pubReadUser = new Grant(); + pubReadUser.grantee = new Grantee(); + pubReadUser.grantee.type = GWConstants.GROUP; + pubReadUser.grantee.uri = GWConstants.AWS_GRANT_URI_ALL_USERS; + pubReadUser.permission = GWConstants.GRANT_READ; + accessControlPolicy.aclList.grants.add(pubReadUser); + + Grant pubWriteUser = new Grant(); + pubWriteUser.grantee = new Grantee(); + pubWriteUser.grantee.type = GWConstants.GROUP; + pubWriteUser.grantee.uri = GWConstants.AWS_GRANT_URI_ALL_USERS; + pubWriteUser.permission = GWConstants.GRANT_WRITE; + accessControlPolicy.aclList.grants.add(pubWriteUser); + } else if (GWConstants.CANNED_ACLS_AUTHENTICATED_READ.equalsIgnoreCase(cannedAcl)) { + if (pabc != null && GWConstants.STRING_TRUE.equalsIgnoreCase(pabc.BlockPublicAcls)) { + logger.info(GWConstants.LOG_ACCESS_DENIED_PUBLIC_ACLS); + throw new GWException(GWErrorCode.ACCESS_DENIED, s3Parameter); + } + Grant priUser = new Grant(); + priUser.grantee = new Grantee(); + priUser.grantee.type = GWConstants.CANONICAL_USER; + priUser.grantee.id = accessControlPolicy.owner.id; + priUser.grantee.displayName = accessControlPolicy.owner.displayName; + priUser.permission = GWConstants.GRANT_FULL_CONTROL; + accessControlPolicy.aclList.grants.add(priUser); + + Grant authReadUser = new Grant(); + authReadUser.grantee = new Grantee(); + authReadUser.grantee.type = GWConstants.GROUP; + authReadUser.grantee.uri = GWConstants.AWS_GRANT_URI_AUTHENTICATED_USERS; + authReadUser.permission = GWConstants.GRANT_READ; + accessControlPolicy.aclList.grants.add(authReadUser); + } else if (GWConstants.CANNED_ACLS_BUCKET_OWNER_READ.equalsIgnoreCase(cannedAcl)) { + Grant priUser = new Grant(); + priUser.grantee = new Grantee(); + priUser.grantee.type = GWConstants.CANONICAL_USER; + priUser.grantee.id = accessControlPolicy.owner.id; + priUser.grantee.displayName = accessControlPolicy.owner.displayName; + priUser.permission = GWConstants.GRANT_FULL_CONTROL; + accessControlPolicy.aclList.grants.add(priUser); + + Grant bucketOwnerReadUser = new Grant(); + bucketOwnerReadUser.grantee = new Grantee(); + bucketOwnerReadUser.grantee.type = GWConstants.CANONICAL_USER; + bucketOwnerReadUser.grantee.id = bucketInfo.getUserId(); + bucketOwnerReadUser.grantee.displayName = bucketInfo.getUserName(); + bucketOwnerReadUser.permission = GWConstants.GRANT_READ; + accessControlPolicy.aclList.grants.add(bucketOwnerReadUser); + } else if (GWConstants.CANNED_ACLS_BUCKET_OWNER_FULL_CONTROL.equalsIgnoreCase(cannedAcl)) { + Grant priUser = new Grant(); + priUser.grantee = new Grantee(); + priUser.grantee.type = GWConstants.CANONICAL_USER; + priUser.grantee.id = accessControlPolicy.owner.id; + priUser.grantee.displayName = accessControlPolicy.owner.displayName; + priUser.permission = GWConstants.GRANT_FULL_CONTROL; + accessControlPolicy.aclList.grants.add(priUser); + + Grant bucketOwnerFullUser = new Grant(); + bucketOwnerFullUser.grantee = new Grantee(); + bucketOwnerFullUser.grantee.type = GWConstants.CANONICAL_USER; + bucketOwnerFullUser.grantee.id = bucketInfo.getUserId(); + bucketOwnerFullUser.grantee.displayName = bucketInfo.getUserName(); + bucketOwnerFullUser.permission = GWConstants.GRANT_FULL_CONTROL; + accessControlPolicy.aclList.grants.add(bucketOwnerFullUser); + } else if (GWConstants.CANNED_ACLS.contains(cannedAcl)) { + logger.error(GWErrorCode.NOT_IMPLEMENTED.getMessage() + GWConstants.LOG_ACCESS_CANNED_ACL, cannedAcl); + throw new GWException(GWErrorCode.NOT_IMPLEMENTED, s3Parameter); + } else { + logger.error(HttpServletResponse.SC_BAD_REQUEST + GWConstants.LOG_ACCESS_PROCESS_FAILED); + throw new GWException(GWErrorCode.BAD_REQUEST, s3Parameter); + } + } + + String getAclHeader = null; + if (!Strings.isNullOrEmpty(getGrantRead)) { + getAclHeader = getGrantRead; + readAclHeader(getGrantRead, GWConstants.GRANT_READ, accessControlPolicy); + } + if (!Strings.isNullOrEmpty(getGrantWrite)) { + getAclHeader = getGrantWrite; + readAclHeader(getGrantWrite, GWConstants.GRANT_WRITE, accessControlPolicy); + } + if (!Strings.isNullOrEmpty(getGrantReadAcp)) { + getAclHeader = getGrantReadAcp; + readAclHeader(getGrantReadAcp, GWConstants.GRANT_READ_ACP, accessControlPolicy); + } + if (!Strings.isNullOrEmpty(getGrantWriteAcp)) { + getAclHeader = getGrantWriteAcp; + readAclHeader(getGrantWriteAcp, GWConstants.GRANT_WRITE_ACP, accessControlPolicy); + } + if (!Strings.isNullOrEmpty(getGrantFullControl)) { + getAclHeader = getGrantFullControl; + readAclHeader(getGrantFullControl, GWConstants.GRANT_FULL_CONTROL, accessControlPolicy); + } + + if (Strings.isNullOrEmpty(aclXml)) { + XmlMapper xmlMapper = new XmlMapper(); + try { + aclXml = xmlMapper.writeValueAsString(accessControlPolicy).replaceAll(GWConstants.WSTXNS, GWConstants.XSI); + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + } else { + logger.info("aclXml : {}", aclXml); + } + + // check user + try { + XmlMapper xmlMapper = new XmlMapper(); + AccessControlPolicy checkAcl = xmlMapper.readValue(aclXml, AccessControlPolicy.class); + if (checkAcl.owner == null) { + if (isAcl) { + throw new GWException(GWErrorCode.MALFORMED_ACL, s3Parameter); + } + checkAcl.owner = new Owner(); + checkAcl.owner.id = s3Parameter.getUser().getUserId(); + checkAcl.owner.displayName = s3Parameter.getUser().getUserName(); + } else if (Strings.isNullOrEmpty(checkAcl.owner.id)) { + if (isAcl) { + throw new GWException(GWErrorCode.MALFORMED_ACL, s3Parameter); + } + checkAcl.owner.id = s3Parameter.getUser().getUserId(); + checkAcl.owner.displayName = s3Parameter.getUser().getUserName(); + } + aclXml = checkAcl.toString(); + logger.info("acl : {}", aclXml); + if (checkAcl.aclList.grants != null) { + if (checkAcl.aclList.grants.size() == 0) { + if (isAcl) { + throw new GWException(GWErrorCode.MALFORMED_ACL, s3Parameter); + } + } + for (Grant user : checkAcl.aclList.grants) { + if (!Strings.isNullOrEmpty(user.grantee.displayName) + && (getAclXml != null || getAclHeader != null) + && S3UserManager.getInstance().getUserByName(user.grantee.displayName) == null) { + logger.info(user.grantee.displayName); + throw new GWException(GWErrorCode.INVALID_ARGUMENT, s3Parameter); + } + + if (!Strings.isNullOrEmpty(user.grantee.id) + && (getAclXml != null || getAclHeader != null) + && S3UserManager.getInstance().getUserById(user.grantee.id) == null) { + logger.info(user.grantee.id); + throw new GWException(GWErrorCode.INVALID_ARGUMENT, s3Parameter); + } + } + } + } catch (JsonProcessingException e) { + PrintStack.logging(logger, e); + throw new GWException(GWErrorCode.SERVER_ERROR, s3Parameter); + } + + return aclXml; + } + + public static String makeAdmAclXml(AccessControlPolicy accessControlPolicy, AccessControlPolicy preAccessControlPolicy, boolean hasKeyWord, String getAclXml, @@ -952,10 +1235,10 @@ public static String makeAclXml(AccessControlPolicy accessControlPolicy, // } // if (!Strings.isNullOrEmpty(user.grantee.id) && GWUtils.getDBInstance().getIdentityByID(user.grantee.id, s3Parameter) == null) { - if (!Strings.isNullOrEmpty(user.grantee.id) && S3UserManager.getInstance().getUserById(user.grantee.id) == null) { - logger.info(user.grantee.id); - throw new GWException(GWErrorCode.INVALID_ARGUMENT, s3Parameter); - } + // if (!Strings.isNullOrEmpty(user.grantee.id) && S3UserManager.getInstance().getUserById(user.grantee.id) == null) { + // logger.info(user.grantee.id); + // throw new GWException(GWErrorCode.INVALID_ARGUMENT, s3Parameter); + // } } } } catch (JsonProcessingException e) { @@ -988,17 +1271,17 @@ public static void initCache(String cacheDisk) { for (Server server : diskpool.getServerList()) { if (GWUtils.getLocalIP().equals(server.getIp())) { for (Disk disk : server.getDiskList()) { - File file = new File(cacheDisk + disk.getPath() + GWConstants.SLASH + GWConstants.OBJ_DIR); + File file = new File(cacheDisk + disk.getPath() + GWConstants.SLASH + Constants.OBJ_DIR); file.mkdirs(); - file = new File(cacheDisk + disk.getPath() + GWConstants.SLASH + GWConstants.TEMP_DIR); + file = new File(cacheDisk + disk.getPath() + GWConstants.SLASH + Constants.TEMP_DIR); file.mkdirs(); - file = new File(cacheDisk + disk.getPath() + GWConstants.SLASH + GWConstants.TRASH_DIR); + file = new File(cacheDisk + disk.getPath() + GWConstants.SLASH + Constants.TRASH_DIR); file.mkdirs(); - file = new File(disk.getPath() + GWConstants.SLASH + GWConstants.OBJ_DIR); + file = new File(disk.getPath() + GWConstants.SLASH + Constants.OBJ_DIR); file.mkdirs(); - file = new File(disk.getPath() + GWConstants.SLASH + GWConstants.TEMP_DIR); + file = new File(disk.getPath() + GWConstants.SLASH + Constants.TEMP_DIR); file.mkdirs(); - file = new File(disk.getPath() + GWConstants.SLASH + GWConstants.TRASH_DIR); + file = new File(disk.getPath() + GWConstants.SLASH + Constants.TRASH_DIR); file.mkdirs(); } } @@ -1030,7 +1313,7 @@ public static CtrCryptoInputStream initCtrDecrypt(FileInputStream in, String cus byte[] iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0x10 }; byte[] key = new byte[32]; - logger.info(customerKey); + logger.info("init ctr decrypt key : {}", customerKey); for (int i = 0; i < 32; i++) { if (i < customerKey.getBytes().length) key[i] = customerKey.getBytes()[i]; @@ -1096,4 +1379,21 @@ public static void ordercombination(int level, int subsetnum, int setnum, String } } } + + /** + * Parse ISO 8601 timestamp into seconds since 1970. + * + * @throws GWException + */ + public static long parseRetentionTimeExpire(String date, S3Parameter s3Parameter) throws GWException { + SimpleDateFormat formatter = new SimpleDateFormat(GWConstants.ISO_8601_TIME_FORMAT_MILI); + formatter.setTimeZone(TimeZone.getTimeZone(GWConstants.UTC)); + logger.debug(GWConstants.LOG_8601_DATE, date); + try { + return formatter.parse(date).getTime() / 1000; + } catch (ParseException pe) { + PrintStack.logging(logger, pe); + throw new GWException(GWErrorCode.BAD_REQUEST, s3Parameter); + } + } } \ No newline at end of file diff --git a/core/src/com/pspace/ifs/ksan/gw/utils/S3UserManager.java b/core/src/com/pspace/ifs/ksan/gw/utils/S3UserManager.java index 3b7a0c35..09458234 100644 --- a/core/src/com/pspace/ifs/ksan/gw/utils/S3UserManager.java +++ b/core/src/com/pspace/ifs/ksan/gw/utils/S3UserManager.java @@ -35,6 +35,10 @@ private S3UserManager() { userSet = new HashSet(); } + public Set getUserSet() { + return userSet; + } + public void addUser(S3User user) { userSet.add(user); } diff --git a/core/src/com/pspace/ifs/ksan/libs/Constants.java b/core/src/com/pspace/ifs/ksan/libs/Constants.java index 4650f4eb..5dcaf4e0 100644 --- a/core/src/com/pspace/ifs/ksan/libs/Constants.java +++ b/core/src/com/pspace/ifs/ksan/libs/Constants.java @@ -13,6 +13,23 @@ public class Constants { public static final int MAXBUFSIZE = 524288; // 512 * 1024 + + public static final char CHAR_SLASH = '/'; + public static final String SLASH = "/"; + public static final String UNDERSCORE = "_"; + public static final String DASH = "-"; + public static final String POINT = "."; + public static final String SPACE = " "; + + public static final String OBJ_DIR = "obj"; + public static final String TEMP_DIR = "temp"; + public static final String TEMP_COMPLETE_DIR = "temp/complete"; + public static final String TEMP_COPY_DIR = "temp/copy"; + public static final String TRASH_DIR = "trash"; + public static final String EC_DIR = "ec"; + + public static final String VERSIONING_DISABLE_TAIL = "null"; + public static final String COLON = ":"; public static final String CHARSET_UTF_8 = "UTF-8"; @@ -44,11 +61,15 @@ public class Constants { public static final String LOG_OSDCLIENT_SOCKET_INFO = "socket : {}"; public static final String LOG_OSDCLIENT_CLOSE_SOCKET_INFO = "close socket : {}"; public static final String LOG_OSDCLIENT_SOCKET_ERROR = "socket error, {}"; - public static final String LOG_OSDCLIENT_HEADER = "get header : {}"; + public static final String LOG_OSDCLIENT_GET_HEADER = "get header : {}"; + public static final String LOG_OSDCLIENT_GET_EC_PART_HEADER = "get ec part header : {}"; public static final String LOG_OSDCLIENT_READ = "read {} bytes"; public static final String LOG_OSDCLIENT_WRITE = "write {} bytes"; public static final String LOG_OSDCLIENT_PUT_HEADER = "put header : {}"; + public static final String LOG_OSDCLIENT_PUT_EC_PART_HEADER = "put ec part header : {}"; public static final String LOG_OSDCLIENT_DELETE_HEADER = "delete header : {}"; + public static final String LOG_OSDCLIENT_DELETE_REPLICA_HEADER = "delete replica header : {}"; + public static final String LOG_OSDCLIENT_DELETE_EC_PART_HEADER = "delete ec part header : {}"; public static final String LOG_OSDCLIENT_COPY_HEADER = "copy header : {}"; public static final String LOG_OSDCLIENT_PART_HEADER = "part header : {}"; public static final String LOG_OSDCLIENT_DELETE_PART_HEADER = "delete part header : {}"; @@ -59,6 +80,21 @@ public class Constants { public static final String LOG_OSDCLIENT_ACTIVATE_SOCKET = "activate socket..."; public static final String LOG_OSDCLIENT_DESACTIVATE_SOCKET = "desactivate socket..."; + public static final String FILE_ATTRIBUTE_REPLICATION = "replica"; + public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID = "diskid"; + public static final String FILE_ATTRUBUTE_REPLICATION_PRIMARY = "pri"; + public static final String FILE_ATTRIBUTE_REPLICATION_REPLICA = "rep"; + public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL = "nul"; + public static final int FILE_ATTRIBUTE_REPLICATION_SIZE = 3; + public static final int FILE_ATTRIBUTE_REPLICATION_DISK_ID_SIZE = 36; + + public static final String ZFEC = "zfec -d "; + public static final String ZFEC_PREFIX_OPTION = " -p "; + public static final String ZFEC_TOTAL_SHARES_OPTION = " -m "; + public static final String ZFEC_REQUIRED_SHARES_OPTION = " -k "; + public static final String ZFEC_SUFFIX = ".fec"; + public static final String ZUNFEC = "zunfec -o "; + private Constants() { throw new IllegalStateException("Utility class"); } diff --git a/core/src/com/pspace/ifs/ksan/libs/DiskManager.java b/core/src/com/pspace/ifs/ksan/libs/DiskManager.java index 9ae41410..9d92287a 100644 --- a/core/src/com/pspace/ifs/ksan/libs/DiskManager.java +++ b/core/src/com/pspace/ifs/ksan/libs/DiskManager.java @@ -49,7 +49,7 @@ public class DiskManager { private static final Logger logger = LoggerFactory.getLogger(DiskManager.class); private List diskPoolList; private String localHost; - private HashMap localDiskInfoMap = new HashMap(); + private HashMap localDiskInfoMap; public static DiskManager getInstance() { return LazyHolder.INSTANCE; @@ -62,6 +62,7 @@ private static class LazyHolder { private DiskManager() { localHost = KsanUtils.getLocalIP(); diskPoolList = new ArrayList(); + localDiskInfoMap = new HashMap(); } public void configure() { @@ -147,6 +148,32 @@ public DiskPool getDiskPool(String id) { return null; } + public int getECM(String diskID) { + for (DiskPool pool : diskPoolList) { + for (Server server : pool.getServerList()) { + for (Disk disk : server.getDiskList()) { + if (diskID.equals(disk.getId())) { + return pool.getErasureCodeM(); + } + } + } + } + return 0; + } + + public int getECK(String diskID) { + for (DiskPool pool : diskPoolList) { + for (Server server : pool.getServerList()) { + for (Disk disk : server.getDiskList()) { + if (diskID.equals(disk.getId())) { + return pool.getErasureCodeK(); + } + } + } + } + return 0; + } + public void saveFile() throws IOException { try { com.google.common.io.Files.createParentDirs(new File(Constants.DISKPOOL_CONF_PATH)); diff --git a/core/src/com/pspace/ifs/ksan/libs/KsanUtils.java b/core/src/com/pspace/ifs/ksan/libs/KsanUtils.java index 3d8c36f6..87c5d6a7 100644 --- a/core/src/com/pspace/ifs/ksan/libs/KsanUtils.java +++ b/core/src/com/pspace/ifs/ksan/libs/KsanUtils.java @@ -15,6 +15,14 @@ import java.io.IOException; import java.net.InetAddress; import java.net.UnknownHostException; +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.StandardCopyOption; +import java.nio.file.attribute.UserDefinedFileAttributeView; +import java.util.UUID; import com.google.common.base.Strings; @@ -72,6 +80,153 @@ public static void retryRenameTo(File srcFile, File destFile) throws IOException } } logger.error(LOG_OSD_SERVER_FAILED_FILE_RENAME, srcFile.getAbsolutePath(), destFile.getAbsolutePath()); + } else { + logger.info("{}, File does not exist. ", srcFile.getAbsolutePath()); } } + + private static String makeDirectoryName(String objId) { + byte[] path = new byte[6]; + byte[] byteObjId = objId.getBytes(); + + path[0] = Constants.CHAR_SLASH; + int index = 1; + + path[index++] = byteObjId[0]; + path[index++] = byteObjId[1]; + path[index++] = Constants.CHAR_SLASH; + path[index++] = byteObjId[2]; + path[index] = byteObjId[3]; + + return new String(path); + } + + public static String makePath(String path, String fileName) { + String fullPath = path + Constants.SLASH + Constants.OBJ_DIR + makeDirectoryName(fileName) + Constants.SLASH + fileName; + return fullPath; + } + + public static String makeObjPath(String path, String objId, String versionId) { + if (Strings.isNullOrEmpty(versionId)) { + versionId = Constants.VERSIONING_DISABLE_TAIL; + } + String fullPath = path + Constants.SLASH + Constants.OBJ_DIR + makeDirectoryName(objId) + Constants.SLASH + objId + Constants.UNDERSCORE + versionId; + return fullPath; + } + + public static String makeTempPath(String path, String objId, String versionId) { + if (Strings.isNullOrEmpty(versionId)) { + versionId = Constants.VERSIONING_DISABLE_TAIL; + } + String uuid = UUID.randomUUID().toString(); + String fullPath = path + Constants.SLASH + Constants.TEMP_DIR + Constants.SLASH + objId + Constants.UNDERSCORE + uuid + Constants.UNDERSCORE + versionId; + return fullPath; + } + + public static String makeTrashPath(String path, String objId, String versionId) { + if (Strings.isNullOrEmpty(versionId)) { + versionId = Constants.VERSIONING_DISABLE_TAIL; + } + String uuid = UUID.randomUUID().toString(); + String fullPath = path + Constants.SLASH + Constants.TRASH_DIR + Constants.SLASH + objId + Constants.UNDERSCORE + versionId + Constants.DASH + uuid; + return fullPath; + } + + public static String makeECPath(String path, String objId, String versionId) { + if (Strings.isNullOrEmpty(versionId)) { + versionId = Constants.VERSIONING_DISABLE_TAIL; + } + String fullPath = path + Constants.SLASH + Constants.EC_DIR + makeDirectoryName(objId) + Constants.SLASH + Constants.POINT + objId + Constants.UNDERSCORE + versionId; + return fullPath; + } + + public static String makeTempPartPath(String path, String objId, String partNumber) { + String fullPath = path + Constants.SLASH + Constants.TEMP_DIR + Constants.SLASH + objId + Constants.UNDERSCORE + partNumber; + return fullPath; + } + + public static String makeTempCompleteMultipartPath(String path, String objId, String versionId) { + if (Strings.isNullOrEmpty(versionId)) { + versionId = Constants.VERSIONING_DISABLE_TAIL; + } + String fullPath = path + Constants.SLASH + Constants.TEMP_COMPLETE_DIR + Constants.SLASH + objId + Constants.UNDERSCORE + versionId; + return fullPath; + } + + public static String makeTempCopyPath(String path, String objId, String versionId) { + if (Strings.isNullOrEmpty(versionId)) { + versionId = Constants.VERSIONING_DISABLE_TAIL; + } + String fullPath = path + Constants.SLASH + Constants.TEMP_COPY_DIR + Constants.SLASH + objId + Constants.UNDERSCORE + versionId; + return fullPath; + } + + public static String makeECDecodePath(String path, String objId, String versionId) { + if (Strings.isNullOrEmpty(versionId)) { + versionId = Constants.VERSIONING_DISABLE_TAIL; + } + String fullPath = path + Constants.SLASH + Constants.EC_DIR + makeDirectoryName(objId) + Constants.SLASH + objId + Constants.UNDERSCORE + versionId; + return fullPath; + } + + public static String makeECDirectory(String fileName, String ecPath) { + String fullPath = ecPath + makeDirectoryName(fileName); + return fullPath; + } + + public static String makeECTempPath(String fileName, String ecPath) { + String fullPath = ecPath + makeDirectoryName(fileName) + Constants.SLASH + Constants.POINT + fileName; + return fullPath; + } + + + public static void setAttributeFileReplication(File file, String replica, String diskID) { + UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); + try { + view.write(Constants.FILE_ATTRIBUTE_REPLICATION, Charset.defaultCharset().encode(replica)); + view.write(Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID, Charset.defaultCharset().encode(diskID)); + logger.debug("Set attribute file replication {}, {}", replica, diskID); + } catch (IOException e) { + PrintStack.logging(logger, e); + } + } + + public static String getAttributeFileReplication(File file) { + UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); + ByteBuffer buf = null; + try { + buf = ByteBuffer.allocateDirect(Constants.FILE_ATTRIBUTE_REPLICATION_SIZE); + view.read(Constants.FILE_ATTRIBUTE_REPLICATION, buf); + buf.flip(); + } catch (IOException e) { + logger.error(e.getMessage()); + } catch (IllegalArgumentException iae) { + logger.error(iae.getMessage()); + } catch (SecurityException e) { + logger.error(e.getMessage()); + } + String replica = Charset.defaultCharset().decode(buf).toString(); + logger.debug("get replica : {}", replica); + return replica; + } + + public static String getAttributeFileReplicaDiskID(File file) { + UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); + ByteBuffer buf = null; + try { + buf = ByteBuffer.allocateDirect(Constants.FILE_ATTRIBUTE_REPLICATION_DISK_ID_SIZE); + view.read(Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID, buf); + buf.flip(); + } catch (IOException e) { + logger.error(e.getMessage()); + } catch (IllegalArgumentException iae) { + logger.error(iae.getMessage()); + } catch (SecurityException e) { + logger.error(e.getMessage()); + } + + String diskID = Charset.defaultCharset().decode(buf).toString(); + logger.debug("file replicaDiskID : {}", diskID); + return diskID; + } } diff --git a/core/src/com/pspace/ifs/ksan/libs/OSDClient.java b/core/src/com/pspace/ifs/ksan/libs/OSDClient.java index b4f66033..44f67ceb 100644 --- a/core/src/com/pspace/ifs/ksan/libs/OSDClient.java +++ b/core/src/com/pspace/ifs/ksan/libs/OSDClient.java @@ -68,12 +68,26 @@ public void getInit(String path, String objId, String versionId, long fileSize, + OsdData.DELIMITER + versionId + OsdData.DELIMITER + sourceRange + OsdData.DELIMITER + key; - logger.debug(Constants.LOG_OSDCLIENT_HEADER, header); + logger.debug(Constants.LOG_OSDCLIENT_GET_HEADER, header); sendHeader(header); this.fileSize = fileSize; byPassOut = out; } + public void getInitWithMD5(String path, String objId, String versionId, long fileSize, String sourceRange, OutputStream out, MessageDigest md5er, String key) throws IOException { + String header = OsdData.GET + + OsdData.DELIMITER + path + + OsdData.DELIMITER + objId + + OsdData.DELIMITER + versionId + + OsdData.DELIMITER + sourceRange + + OsdData.DELIMITER + key; + logger.debug(Constants.LOG_OSDCLIENT_GET_HEADER, header); + sendHeader(header); + this.fileSize = fileSize; + byPassOut = out; + this.md5er = md5er; + } + public long get() throws IOException { byte[] buffer = new byte[Constants.MAXBUFSIZE]; int readByte = Constants.MAXBUFSIZE; @@ -91,6 +105,26 @@ public long get() throws IOException { // byPassOut.close(); logger.info(Constants.LOG_OSDCLIENT_READ, readTotal); return readTotal; + } + + public long getWithMD5() throws IOException { + byte[] buffer = new byte[Constants.MAXBUFSIZE]; + int readByte = Constants.MAXBUFSIZE; + int readLength = 0; + long readTotal = 0L; + + while ((readLength = socket.getInputStream().read(buffer, 0, readByte)) != -1) { + readTotal += readLength; + byPassOut.write(buffer, 0, readLength); + md5er.update(buffer, 0, readLength); + if (readTotal >= fileSize) { + break; + } + } + byPassOut.flush(); + // byPassOut.close(); + logger.info(Constants.LOG_OSDCLIENT_READ, readTotal); + return readTotal; } public void getPartInit(String path, String objId, String partNo, long fileSize, OutputStream out, MessageDigest md5er) throws IOException { @@ -98,7 +132,7 @@ public void getPartInit(String path, String objId, String partNo, long fileSize, + OsdData.DELIMITER + path + OsdData.DELIMITER + objId + OsdData.DELIMITER + partNo; - logger.debug(Constants.LOG_OSDCLIENT_HEADER, header); + logger.debug(Constants.LOG_OSDCLIENT_GET_HEADER, header); sendHeader(header); this.fileSize = fileSize; byPassOut = out; @@ -158,6 +192,20 @@ public void delete(String path, String objId, String versionId) throws IOExcepti sendHeader(header); } + public void deleteReplica(String path) throws IOException { + String header = OsdData.DELETE_REPLICA + + OsdData.DELIMITER + path; + logger.debug(Constants.LOG_OSDCLIENT_DELETE_REPLICA_HEADER, header); + sendHeader(header); + } + + public void deleteECPart(String path) throws IOException { + String header = OsdData.DELETE_EC_PART + + OsdData.DELIMITER + path; + logger.debug(Constants.LOG_OSDCLIENT_DELETE_EC_PART_HEADER, header); + sendHeader(header); + } + public void copy(String srcPath, String srcObjId, String srcVersionId, String destPath, String destObjId, String destVersionId, String replication, String replicaDiskID) throws IOException { String header = OsdData.COPY + OsdData.DELIMITER + srcPath @@ -196,27 +244,30 @@ public void deletePart(String path, String objId, String partNo) throws IOExcept sendHeader(header); } - public OsdData partCopy(String srcPath, String srcObjId, String srcVersionId, String copySourceRange, String destPath, String destObjId, String partNo) throws IOException { + public OsdData partCopy(String srcDiskId, String srcObjId, String srcVersionId, String srcLength, String copySourceRange, String destPath, String destObjId, String partNo) throws IOException { String header = OsdData.PART_COPY - + OsdData.DELIMITER + srcPath + + OsdData.DELIMITER + srcDiskId + OsdData.DELIMITER + srcObjId + OsdData.DELIMITER + srcVersionId + OsdData.DELIMITER + destPath + OsdData.DELIMITER + destObjId + OsdData.DELIMITER + partNo - + OsdData.DELIMITER + copySourceRange; + + OsdData.DELIMITER + copySourceRange + + OsdData.DELIMITER + srcLength; logger.debug(Constants.LOG_OSDCLIENT_PART_COPY_HEADER, header); sendHeader(header); return receiveData(); } - public OsdData completeMultipart(String path, String objId, String versionId, String key, String partNos) throws IOException { + public OsdData completeMultipart(String path, String objId, String versionId, String key, String replication, String replicaDiskID, String partNos) throws IOException { String header = OsdData.COMPLETE_MULTIPART + OsdData.DELIMITER + path + OsdData.DELIMITER + objId + OsdData.DELIMITER + versionId + OsdData.DELIMITER + key + + OsdData.DELIMITER + replication + + OsdData.DELIMITER + replicaDiskID + OsdData.DELIMITER + partNos; logger.debug(Constants.LOG_OSDCLIENT_COMPLETE_MULTIPART_HEADER, header); sendHeader(header); @@ -233,6 +284,46 @@ public void abortMultipart(String path, String objId, String partNos) throws IOE sendHeader(header); } + public void getECPartInit(String path, OutputStream out) throws IOException { + String header = OsdData.GET_EC_PART + + OsdData.DELIMITER + path; + logger.debug(Constants.LOG_OSDCLIENT_GET_EC_PART_HEADER, header); + sendHeader(header); + byPassOut = out; + } + + public long getECPart() throws IOException { + byte[] buffer = new byte[Constants.MAXBUFSIZE]; + int readByte = Constants.MAXBUFSIZE; + int readLength = 0; + long readTotal = 0L; + + while ((readLength = socket.getInputStream().read(buffer, 0, readByte)) != -1) { + readTotal += readLength; + byPassOut.write(buffer, 0, readLength); + } + byPassOut.flush(); + // byPassOut.close(); + logger.info(Constants.LOG_OSDCLIENT_READ, readTotal); + return readTotal; + } + + public void putECPartInit(String path, long length) throws IOException { + String header = OsdData.PUT_EC_PART + + OsdData.DELIMITER + path + + OsdData.DELIMITER + String.valueOf(length); + logger.debug(Constants.LOG_OSDCLIENT_PUT_EC_PART_HEADER, header); + sendHeader(header); + } + + public void putECPart(byte[] buffer, int offset, int length) throws IOException { + socket.getOutputStream().write(buffer, offset, length); + } + + public void putECPartFlush() throws IOException { + socket.getOutputStream().flush(); + } + private void sendHeader(String header) throws IOException { byte[] buffer = header.getBytes(Constants.CHARSET_UTF_8); String strLength = Integer.toString(buffer.length); @@ -248,11 +339,19 @@ private void sendHeader(String header) throws IOException { } private OsdData receiveData() throws IOException { - DataInputStream si = new DataInputStream(socket.getInputStream()); - int size = si.readInt(); - byte[] buffer = new byte[size]; - si.read(buffer, 0, size); - String result = new String(buffer, 0, size); + int length = socket.getInputStream().read(); + if (length == -1) { + logger.info("socket {} EOF ...", socket.getRemoteSocketAddress().toString()); + return null; + } + byte[] lengthBuffer = new byte[length]; + socket.getInputStream().read(lengthBuffer, 0, length); + String strLength = new String(lengthBuffer); + + length = Integer.parseInt(strLength); + byte[] buffer = new byte[length]; + socket.getInputStream().read(buffer, 0, length); + String result = new String(buffer, 0, length); String[] ArrayResult = result.split(Constants.COLON, -1); OsdData data = new OsdData(); @@ -266,6 +365,25 @@ private OsdData receiveData() throws IOException { } return null; + + // DataInputStream si = new DataInputStream(socket.getInputStream()); + // int size = si.readInt(); + // byte[] buffer = new byte[size]; + // si.read(buffer, 0, size); + // String result = new String(buffer, 0, size); + // String[] ArrayResult = result.split(Constants.COLON, -1); + + // OsdData data = new OsdData(); + // switch (ArrayResult[0]) { + // case OsdData.FILE: + // data.setETag(ArrayResult[1]); + // data.setFileSize(Long.parseLong(ArrayResult[2])); + // return data; + // default: + // logger.error(Constants.LOG_OSDCLIENT_UNKNOWN_RESULT, ArrayResult[1]); + // } + + // return null; } public String getHost() { @@ -286,5 +404,32 @@ public void activate() { public void desactivate() { logger.debug(Constants.LOG_OSDCLIENT_DESACTIVATE_SOCKET); } + + public void getECInit(String path, String objId, String versionId, long fileSize, String sourceRange, OutputStream out, String key) throws IOException { + String header = OsdData.GET_EC_PART + + OsdData.DELIMITER + path + + OsdData.DELIMITER + objId + + OsdData.DELIMITER + versionId + + OsdData.DELIMITER + sourceRange + + OsdData.DELIMITER + key; + logger.debug(Constants.LOG_OSDCLIENT_GET_HEADER, header); + sendHeader(header); + this.fileSize = fileSize; + byPassOut = out; + } + + public void putECInit(String path, String objId, String versionId, long length, String replication, String replicaDiskID, String key, String mode) throws IOException { + String header = OsdData.PUT_EC_PART + + OsdData.DELIMITER + path + + OsdData.DELIMITER + objId + + OsdData.DELIMITER + versionId + + OsdData.DELIMITER + String.valueOf(length) + + OsdData.DELIMITER + replication + + OsdData.DELIMITER + replicaDiskID + + OsdData.DELIMITER + key + + OsdData.DELIMITER + mode; + logger.debug(Constants.LOG_OSDCLIENT_PUT_HEADER, header); + sendHeader(header); + } } diff --git a/core/src/com/pspace/ifs/ksan/libs/data/ECPart.java b/core/src/com/pspace/ifs/ksan/libs/data/ECPart.java new file mode 100644 index 00000000..a880ccbc --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/libs/data/ECPart.java @@ -0,0 +1,39 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +package com.pspace.ifs.ksan.libs.data; + +public class ECPart { + private String serverIP; + private String diskPath; + private boolean isProcessed; + + public ECPart(String serverIP, String diskPath, boolean isProcessed) { + this.serverIP = serverIP; + this.diskPath = diskPath; + this.isProcessed = isProcessed; + } + + public String getServerIP() { + return serverIP; + } + + public String getDiskPath() { + return diskPath; + } + + public void setProcessed(boolean isProcessed) { + this.isProcessed = isProcessed; + } + + public boolean isProcessed() { + return isProcessed; + } +} diff --git a/core/src/com/pspace/ifs/ksan/libs/data/OsdData.java b/core/src/com/pspace/ifs/ksan/libs/data/OsdData.java index 8d7e14ce..edbf7854 100644 --- a/core/src/com/pspace/ifs/ksan/libs/data/OsdData.java +++ b/core/src/com/pspace/ifs/ksan/libs/data/OsdData.java @@ -14,7 +14,7 @@ public class OsdData { public static final String STOP = "ST"; public static final String GET = "GE"; public static final String PUT = "PU"; - public static final String DELETE = "DE"; + public static final String DELETE = "DT"; public static final String DELETE_REPLICA = "DR"; public static final String COPY = "CO"; public static final String PART = "PA"; @@ -24,6 +24,9 @@ public class OsdData { public static final String GET_PART = "GP"; public static final String DELETE_PART = "DP"; public static final String FILE = "FILE"; + public static final String GET_EC_PART = "GC"; + public static final String PUT_EC_PART = "PE"; + public static final String DELETE_EC_PART = "DE"; public static final String DELIMITER = ":"; public static final int INDICATOR_SIZE = 2; @@ -37,6 +40,7 @@ public class OsdData { public static final int DEST_VERSIONID_INDEX = 6; public static final int DEST_PARTNO_INDEX = 6; public static final int SRC_RANAGE_INDEX = 7; + public static final int SRC_LENGTH_INDEX = 8; public static final int SOURCE_PATH_INDEX = 1; public static final int SOURCE_OBJID_INDEX = 2; public static final int TARGET_PATH_INDEX = 3; @@ -50,7 +54,9 @@ public class OsdData { public static final int PUT_MODE_INDEX = 8; public static final int PARTNO_INDEX = 3; public static final int COMPLETE_MULTIPART_KEY_INDEX = 4; - public static final int COMPLETE_MULTIPART_PARTNOS_INDEX = 5; + public static final int COMPLETE_MULTIPART_REPLICATION_INDEX = 5; + public static final int COMPLETE_MULTIPART_REPLICA_DISKID_INDEX = 6; + public static final int COMPLETE_MULTIPART_PARTNOS_INDEX = 7; public static final int ABORT_MULTIPART_PARTNOS = 3; public static final int PART_NO_INDEX = 3; public static final int PARTNOS_INDEX = 3; @@ -59,6 +65,7 @@ public class OsdData { public static final int PART_COPY_LENGTH_INDEX = 8; public static final int COPY_REPLICATION_INDED = 7; public static final int COPY_REPLICA_DISK_ID_INDEX = 8; + public static final int PUT_EC_LENGTH_INDEX = 2; private String ETag; private long fileSize; diff --git a/core/src/com/pspace/ifs/ksan/libs/disk/DiskPool.java b/core/src/com/pspace/ifs/ksan/libs/disk/DiskPool.java index e48e81d6..ff2e02c1 100644 --- a/core/src/com/pspace/ifs/ksan/libs/disk/DiskPool.java +++ b/core/src/com/pspace/ifs/ksan/libs/disk/DiskPool.java @@ -22,13 +22,19 @@ public class DiskPool { public static final String REPLICATION_TYPE_ONE_PLUS_ZERO = "OnePlusZero"; public static final String REPLICATION_TYPE_ONE_PLUS_ONE = "OnePlusOne"; public static final String REPLICATION_TYPE_ONE_PLUS_TWO = "OnePlusTwo"; + public static final String REPLICATION_TYPE_ERASURE_CODE = "ErasureCode"; public static final String SERVERS = "Servers"; + public static final String EC = "EC"; + public static final String EC_M = "M"; // the number of coding chunks + public static final String EC_K = "K"; // the number of data chunks private String id; private String name; private String classTypeId; // "STANDARD", "ARCHIVE" - private String replicationType; // "OnePlusZero", "OnePlusOne", "OnePlusTwo" + private String replicationType; // "OnePlusZero", "OnePlusOne", "OnePlusTwo", "ErasureCode" private int replicaCount; + private int ecM; + private int ecK; private List serverList; public DiskPool() { @@ -36,6 +42,8 @@ public DiskPool() { name = ""; classTypeId = ""; replicationType = ""; + ecM = 0; + ecK = 0; serverList = new ArrayList(); } @@ -44,6 +52,18 @@ public DiskPool(String id, String name, String classTypeId, String replicationTy this.name = name; this.classTypeId = classTypeId; this.replicationType = replicationType; + ecM = 0; + ecK = 0; + serverList = new ArrayList(); + } + + public DiskPool(String id, String name, String classTypeId, String replicationType, int ecM, int ecK) { + this.id = id; + this.name = name; + this.classTypeId = classTypeId; + this.replicationType = replicationType; + this.ecM = ecM; + this.ecK = ecK; serverList = new ArrayList(); } @@ -84,6 +104,8 @@ public void setReplicationType(String replicationType) { replicaCount = 1; } else if (this.replicationType.equalsIgnoreCase(REPLICATION_TYPE_ONE_PLUS_TWO)) { replicaCount = 3; + } else if (this.replicationType.equalsIgnoreCase(REPLICATION_TYPE_ERASURE_CODE)) { + replicaCount = 2; } else { replicaCount = 0; } @@ -105,4 +127,12 @@ public void addServer(Server server) { public int getReplicaCount() { return replicaCount; } + + public int getErasureCodeM() { + return ecM; + } + + public int getErasureCodeK() { + return ecK; + } } diff --git a/core/src/com/pspace/ifs/ksan/libs/identity/S3Metadata.java b/core/src/com/pspace/ifs/ksan/libs/identity/S3Metadata.java index b93d0bd3..2d597aee 100644 --- a/core/src/com/pspace/ifs/ksan/libs/identity/S3Metadata.java +++ b/core/src/com/pspace/ifs/ksan/libs/identity/S3Metadata.java @@ -91,6 +91,9 @@ public S3Metadata() { versionId = ""; uploadId = ""; partNumber = 0; + lockMode = ""; + lockExpires = ""; + legalHold = ""; } public Map getUserMetadataMap() { @@ -110,7 +113,7 @@ public String getUserMetadata(String key) { } public String getStorageType() { - return Strings.nullToEmpty(storageType); + return storageType; } public void setStorageType(String storageType) { @@ -118,7 +121,7 @@ public void setStorageType(String storageType) { } public String getName() { - return Strings.nullToEmpty(name); + return name; } public void setName(String name) { @@ -126,7 +129,7 @@ public void setName(String name) { } public String getUri() { - return Strings.nullToEmpty(uri); + return uri; } public void setUri(String uri) { @@ -134,7 +137,7 @@ public void setUri(String uri) { } public String getETag() { - return Strings.nullToEmpty(eTag); + return eTag; } public void setETag(String eTag) { @@ -174,7 +177,7 @@ public void setSize(Long size) { } public String getTier() { - return Strings.nullToEmpty(tier); + return tier; } public void setTier(String tier) { @@ -182,7 +185,7 @@ public void setTier(String tier) { } public String getCacheControl() { - return Strings.nullToEmpty(cacheControl); + return cacheControl; } public void setCacheControl(String cacheControl) { @@ -190,7 +193,7 @@ public void setCacheControl(String cacheControl) { } public String getReadEncryption() { - return Strings.nullToEmpty(readEncryption); + return readEncryption; } public void setReadEncryption(String readEncryption) { @@ -198,7 +201,7 @@ public void setReadEncryption(String readEncryption) { } public String getReadEncryptionSize() { - return Strings.nullToEmpty(readEncryptionSize); + return readEncryptionSize; } public void setReadEncryptionSize(String readEncryptionSize) { @@ -206,7 +209,7 @@ public void setReadEncryptionSize(String readEncryptionSize) { } public String getCustomerAlgorithm() { - return Strings.nullToEmpty(customerAlgorithm); + return customerAlgorithm; } public void setCustomerAlgorithm(String customerAlgorithm) { @@ -214,7 +217,7 @@ public void setCustomerAlgorithm(String customerAlgorithm) { } public String getCustomerKey() { - return Strings.nullToEmpty(customerKey); + return customerKey; } public void setCustomerKey(String customerKey) { @@ -222,7 +225,7 @@ public void setCustomerKey(String customerKey) { } public String getCustomerKeyMD5() { - return Strings.nullToEmpty(customerKeyMD5); + return customerKeyMD5; } public void setCustomerKeyMD5(String customerKeyMD5) { @@ -230,7 +233,7 @@ public void setCustomerKeyMD5(String customerKeyMD5) { } public String getServersideEncryption() { - return Strings.nullToEmpty(serversideEncryption); + return serversideEncryption; } public void setServersideEncryption(String serversideEncryption) { @@ -238,7 +241,7 @@ public void setServersideEncryption(String serversideEncryption) { } public String getDeleteMarker() { - return Strings.nullToEmpty(deleteMarker); + return deleteMarker; } public void setDeleteMarker(String deleteMarker) { @@ -246,7 +249,7 @@ public void setDeleteMarker(String deleteMarker) { } public String getIsLatest() { - return Strings.nullToEmpty(isLatest); + return isLatest; } public void setIsLatest(String isLatest) { @@ -262,7 +265,7 @@ public void setContentLength(Long contentLength) { } public String getContentDisposition() { - return Strings.nullToEmpty(contentDisposition); + return contentDisposition; } public void setContentDisposition(String contentDisposition) { @@ -270,7 +273,7 @@ public void setContentDisposition(String contentDisposition) { } public String getContentEncoding() { - return Strings.nullToEmpty(contentEncoding); + return contentEncoding; } public void setContentEncoding(String contentEncoding) { @@ -278,7 +281,7 @@ public void setContentEncoding(String contentEncoding) { } public String getContentType() { - return Strings.nullToEmpty(contentType); + return contentType; } public void setContentType(String contentType) { @@ -286,7 +289,7 @@ public void setContentType(String contentType) { } public String getOwnerId() { - return Strings.nullToEmpty(ownerId); + return ownerId; } public void setOwnerId(String ownerId) { @@ -294,7 +297,7 @@ public void setOwnerId(String ownerId) { } public String getOwnerName() { - return Strings.nullToEmpty(ownerName); + return ownerName; } public void setOwnerName(String ownerName) { @@ -302,7 +305,7 @@ public void setOwnerName(String ownerName) { } public String getContentLanguage() { - return Strings.nullToEmpty(contentLanguage); + return contentLanguage; } public void setContentLanguage(String contentLanguage) { @@ -318,7 +321,7 @@ public void setExpires(Date expires) { } public String getTaggingCount() { - return Strings.nullToEmpty(taggingCount); + return taggingCount; } public void setTaggingCount(String taggingCount) { @@ -334,7 +337,7 @@ public void setDecodedContentLength(long decodedContentLength) { } public String getContentMD5() { - return Strings.nullToEmpty(contentMD5); + return contentMD5; } public void setContentMD5(String contentMD5) { @@ -342,7 +345,7 @@ public void setContentMD5(String contentMD5) { } public String getEncryption() { - return Strings.nullToEmpty(encryption); + return encryption; } public void setEncryption(String encryption) { @@ -350,7 +353,7 @@ public void setEncryption(String encryption) { } public String getVersionId() { - return Strings.nullToEmpty(versionId); + return versionId; } public void setVersionId(String versionId) { @@ -358,7 +361,7 @@ public void setVersionId(String versionId) { } public String getUploadId() { - return Strings.nullToEmpty(uploadId); + return uploadId; } public void setUploadId(String uploadId) { diff --git a/core/src/com/pspace/ifs/ksan/libs/mq/MQReceiver.java b/core/src/com/pspace/ifs/ksan/libs/mq/MQReceiver.java index 566ac667..b6193720 100644 --- a/core/src/com/pspace/ifs/ksan/libs/mq/MQReceiver.java +++ b/core/src/com/pspace/ifs/ksan/libs/mq/MQReceiver.java @@ -20,15 +20,15 @@ public MQReceiver(String host, String qname, boolean qdurablity, MQCallback call super(host, qname, qdurablity, callback); } - public MQReceiver(String host, String qname, String exchangeName, boolean queeuDurableity, String exchangeOption, String routingKey, MQCallback callback) + public MQReceiver(String host, String qname, String exchangeName, boolean singleActiveConsumer, String exchangeOption, String routingKey, MQCallback callback) throws Exception{ - super(host, qname, exchangeName, queeuDurableity, exchangeOption, routingKey, callback); + super(host, qname, exchangeName, singleActiveConsumer, exchangeOption, routingKey, callback); } - public MQReceiver(String host, int port, String username, String password, String qname, String exchangeName, boolean queeuDurableity, String exchangeOption, String routingKey, MQCallback callback) + public MQReceiver(String host, int port, String username, String password, String qname, String exchangeName, boolean singleActiveConsumer, String exchangeOption, String routingKey, MQCallback callback) throws Exception{ - super(host, port, username, password, qname, exchangeName, queeuDurableity, exchangeOption, routingKey, callback); + super(host, port, username, password, qname, exchangeName, singleActiveConsumer, exchangeOption, routingKey, callback); } } diff --git a/core/src/com/pspace/ifs/ksan/libs/mq/MessageQ.java b/core/src/com/pspace/ifs/ksan/libs/mq/MessageQ.java index f583e2f7..6a3cbadb 100644 --- a/core/src/com/pspace/ifs/ksan/libs/mq/MessageQ.java +++ b/core/src/com/pspace/ifs/ksan/libs/mq/MessageQ.java @@ -47,12 +47,12 @@ public abstract class MessageQ{ private Logger logger; // for 1-1 sender with default username and password - public MessageQ(String host, String qname, boolean qdurablity) throws Exception{ - this(host, 0, "guest", "guest", qname, qdurablity); + public MessageQ(String host, String qname, boolean singleActiveConsumer) throws Exception{ + this(host, 0, "guest", "guest", qname, singleActiveConsumer); } // for 1-1 sender - public MessageQ(String host, int port, String username, String password, String qname, boolean qdurablity) throws Exception{ + public MessageQ(String host, int port, String username, String password, String qname, boolean singleActiveConsumer) throws Exception{ logger = LoggerFactory.getLogger(MessageQ.class); this.message = ""; this.host = host; @@ -63,15 +63,15 @@ public MessageQ(String host, int port, String username, String password, String this.password = password; this.port = 0; this.connect(); - this.createQ(qdurablity); + this.createQ(singleActiveConsumer); } // for 1-1 receiver with default username and password - public MessageQ(String host, String qname, boolean qdurablity, MQCallback callback) throws Exception{ - this(host, 0, "guest", "guest", qname, qdurablity, callback); + public MessageQ(String host, String qname, boolean singleActiveConsumer, MQCallback callback) throws Exception{ + this(host, 0, "guest", "guest", qname, singleActiveConsumer, callback); } // for 1-1 receiver - public MessageQ(String host, int port, String username, String password, String qname, boolean qdurablity, MQCallback callback) + public MessageQ(String host, int port, String username, String password, String qname, boolean singleActiveConsumer, MQCallback callback) throws Exception{ logger = LoggerFactory.getLogger(MessageQ.class); this.message = ""; @@ -83,7 +83,7 @@ public MessageQ(String host, int port, String username, String password, String this.password = password; this.port = port; this.connect(); - this.createQ(qdurablity); + this.createQ(singleActiveConsumer); this.callback = callback; if (this.callback != null) this.readFromQueue(); @@ -115,13 +115,13 @@ public MessageQ(String host, int port, String username, String password, String } // for 1-n receiver with default username and password - public MessageQ(String host, String qname, String exchangeName, boolean qdurablity, + public MessageQ(String host, String qname, String exchangeName, boolean singleActiveConsumer, String exchangeOption, String routingKey, MQCallback callback) throws Exception{ - this(host, 0, "guest", "guest", qname, exchangeName, qdurablity, exchangeOption, routingKey, callback); + this(host, 0, "guest", "guest", qname, exchangeName, singleActiveConsumer, exchangeOption, routingKey, callback); } // for 1-n receiver - public MessageQ(String host, int port, String username, String password, String qname, String exchangeName, boolean qdurablity, + public MessageQ(String host, int port, String username, String password, String qname, String exchangeName, boolean singleActiveConsumer, String exchangeOption, String routingKey, MQCallback callback) throws Exception{ logger = LoggerFactory.getLogger(MessageQ.class); this.message = ""; @@ -134,7 +134,7 @@ public MessageQ(String host, int port, String username, String password, String this.password = password; this.port = port; this.connect(); - this.createQ(qdurablity); + this.createQ(singleActiveConsumer); this.callback = callback; if (this.callback != null) @@ -170,14 +170,15 @@ private int connect() throws IOException, TimeoutException { return 0; } - private Map getQuorumQueueMap(){ + private Map getQuorumQueueMap(boolean singleActiveConsumer){ Map map = new HashMap<>(); map.put("x-queue-type", "quorum"); - map.put("x-single-active-consumer", true); + if (singleActiveConsumer) + map.put("x-single-active-consumer", true); return map; } - private int createQ(boolean durable) throws Exception{ + private int createQ(boolean singleActiveConsumer) throws Exception{ if (!this.qname.isEmpty()){ try{ this.channel.queueDeclarePassive(this.qname); @@ -185,21 +186,21 @@ private int createQ(boolean durable) throws Exception{ if (!this.channel.isOpen()) this.connect(); - this.channel.queueDeclare(this.qname, true, false, false, getQuorumQueueMap()); + this.channel.queueDeclare(this.qname, true, false, false, getQuorumQueueMap(singleActiveConsumer)); //this.channel.queueDeclare(this.qname, durable, false, false, null); } } return 0; } - private String createQurumQueue(String replyQueue) throws IOException, TimeoutException { + private String createQurumQueue(String replyQueue, boolean singleActiveConsumer) throws IOException, TimeoutException { try{ return channel.queueDeclarePassive(replyQueue).getQueue(); } catch(IOException ex){ if (!this.channel.isOpen()) this.connect(); - return channel.queueDeclare(replyQueue, true, false, false, getQuorumQueueMap()).getQueue(); + return channel.queueDeclare(replyQueue, true, false, false, getQuorumQueueMap(singleActiveConsumer)).getQueue(); } } @@ -247,7 +248,7 @@ public int sendToQueue(String mesg) throws Exception{ if (qname.isEmpty()){ final String corrId = UUID.randomUUID().toString(); - replyQueueName = createQurumQueue("replyQ"); + replyQueueName = createQurumQueue("replyQ", false); } BasicProperties props = new BasicProperties @@ -269,7 +270,7 @@ public int sendToExchange(String mesg, String routingKey) throws Exception{ this.connect(); final String corrId = UUID.randomUUID().toString(); - String replyQueueName = createQurumQueue("replyQ"); + String replyQueueName = createQurumQueue("replyQ", false); BasicProperties props = new BasicProperties .Builder() @@ -283,7 +284,7 @@ public int sendToExchange(String mesg, String routingKey) throws Exception{ public String sendToExchangeWithResponse(String mesg, String routingKey, int timeoutInMilliSec) throws IOException, InterruptedException, TimeoutException{ final String corrId = UUID.randomUUID().toString(); - String replyQueueName = createQurumQueue("replyQ"); + String replyQueueName = createQurumQueue("replyQ", false); BasicProperties props = new BasicProperties .Builder() @@ -303,7 +304,7 @@ public String sendToExchangeWithResponse(String mesg, String routingKey, int tim public String sendToQueueWithResponse(String mesg, int timeoutInMilliSec) throws IOException, InterruptedException, TimeoutException{ final String corrId = UUID.randomUUID().toString(); - String replyQueueName = createQurumQueue("replyQ"); + String replyQueueName = createQurumQueue("replyQ", false); BasicProperties props = new BasicProperties .Builder() .correlationId(corrId) diff --git a/core/src/com/pspace/ifs/ksan/objmanager/Bucket.java b/core/src/com/pspace/ifs/ksan/objmanager/Bucket.java index 54c14e13..0ebe762d 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/Bucket.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/Bucket.java @@ -44,6 +44,7 @@ public class Bucket { private int replicaCount; private long usedSpace; // new private long fileCount; // new + private boolean objTagIndexEnabled; public Bucket(){ name = ""; @@ -62,6 +63,7 @@ public Bucket(){ logging = ""; usedSpace = 0; fileCount = 0; + objTagIndexEnabled = false; } public Bucket(String name, String id, String diskPoolId){ @@ -81,6 +83,7 @@ public Bucket(String name, String id, String diskPoolId){ logging = ""; usedSpace = 0; fileCount = 0; + objTagIndexEnabled = false; } public Bucket(String name, String id, String diskPoolId, String versioning, String mfaDelete, String userId, String acl, Date createTime){ @@ -103,6 +106,7 @@ public Bucket(String name, String id, String diskPoolId, String versioning, Stri logging = ""; usedSpace = 0; fileCount = 0; + objTagIndexEnabled = false; } private void setHashOfName() { @@ -204,7 +208,11 @@ public void setReplicaCount(int replicaCount){ } public void setLogging(String logging){ - this.tagging = logging; + this.logging = logging; + } + + public void setObjectTagIndexEnabled(boolean enabled){ + objTagIndexEnabled = enabled; } public String getName(){ @@ -297,6 +305,10 @@ public String getLogging(){ return this.logging; } + public boolean isObjectTagIndexEnabled(){ + return objTagIndexEnabled; + } + @Override public String toString(){ SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); diff --git a/core/src/com/pspace/ifs/ksan/objmanager/BucketManager.java b/core/src/com/pspace/ifs/ksan/objmanager/BucketManager.java index 261022a1..0afebee1 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/BucketManager.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/BucketManager.java @@ -222,4 +222,11 @@ public void updateBucketLogging(String bucketName, String logging) throws Resour public boolean isBucketDeleted(String bucketName) throws SQLException { return dbm.isBucketDeleted(bucketName); } + + public void updateBucketTagsIndexing(String bucketName, boolean isEnabled) throws SQLException, ResourceNotFoundException { + Bucket bt = getBucket(bucketName); + bt.setObjectTagIndexEnabled(isEnabled); + dbm.updateBucketObjTagIndexing(bt); + obmCache.updateBucketInCache(bt); + } } diff --git a/core/src/com/pspace/ifs/ksan/objmanager/DataRepository.java b/core/src/com/pspace/ifs/ksan/objmanager/DataRepository.java index d4e9cece..621e9860 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/DataRepository.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/DataRepository.java @@ -69,6 +69,7 @@ public interface DataRepository { public void updateBucketUsedSpace(Bucket bt, long size) throws SQLException ; //public void updateBucketFileCount(Bucket bt) throws SQLException ; public void updateBucketLogging(Bucket bt) throws SQLException; + public void updateBucketObjTagIndexing(Bucket bt) throws SQLException; // for multipart upload public int insertMultipartUpload(String bucket, String objkey, String uploadid, int partNo, String acl, String meta, String etag, long size, String pdiskid) throws SQLException; @@ -97,4 +98,12 @@ public interface DataRepository { public List selectAllFailedLifeCycle() throws SQLException; public int deleteLifeCycle(LifeCycle lc) throws SQLException; public int deleteFailedLifeCycle(LifeCycle lc) throws SQLException; + + // for object tags indexing + public List listObjectWithTags(String bucketName, Object query, int maxKeys) throws SQLException; + + // for restore object + public int insertRestoreObjectRequest(String bucketName, String key, String objId, String versionId, String request) throws SQLException; + public String getRestoreObjectRequest(String bucketName, String objId, String versionId) throws SQLException; + public void deleteRestoreObjectRequest(String bucketName, String objId, String versionId) throws SQLException; } diff --git a/core/src/com/pspace/ifs/ksan/objmanager/DataRepositoryQuery.java b/core/src/com/pspace/ifs/ksan/objmanager/DataRepositoryQuery.java index 3cac3c9a..d9f7cdc8 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/DataRepositoryQuery.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/DataRepositoryQuery.java @@ -53,14 +53,14 @@ public final class DataRepositoryQuery { + "encryption TEXT, objectlock TEXT, policy TEXT, " + "versioning VARCHAR(50), MfaDelete VARCHAR(50), " + "createTime timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, " - + "replicaCount INT DEFAULT 2, " + + "replicaCount INT DEFAULT 2, objTagIndexing BOOLEAN default false, " + "usedSpace BIGINT NOT NULL DEFAULT 0, fileCount BIGINT NOT NULL DEFAULT 0, " + "PRIMARY KEY(id)) ENGINE=INNODB DEFAULT CHARSET=UTF8mb4 COLLATE=utf8mb4_unicode_ci;"; - public static String insertBucketQuery = "INSERT INTO BUCKETS(name, id, diskPoolId, userName, userId, acl, encryption, objectlock, replicaCount) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?)"; + public static String insertBucketQuery = "INSERT INTO BUCKETS(name, id, diskPoolId, userName, userId, acl, encryption, objectlock, replicaCount, objTagIndexing) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"; public static String deleteBucketQuery = "DELETE FROM BUCKETS WHERE id=?"; - public static String selectBucketQuery = "SELECT name, id, diskPoolId, versioning, MfaDelete, userName, userId, acl, web, cors, lifecycle, access, tagging, replication, encryption, objectlock, policy, createTime, replicaCount, usedSpace, fileCount, logging FROM BUCKETS WHERE id=?"; - public static String selectAllBucketQuery = "SELECT name, id, diskPoolId, versioning, MfaDelete, userName, userId, acl, web, cors, lifecycle, access, tagging, replication, encryption, objectlock, policy, createTime, replicaCount, usedSpace, fileCount, logging FROM BUCKETS"; + public static String selectBucketQuery = "SELECT name, id, diskPoolId, versioning, MfaDelete, userName, userId, acl, web, cors, lifecycle, access, tagging, replication, encryption, objectlock, policy, createTime, replicaCount, usedSpace, fileCount, logging, objTagIndexing FROM BUCKETS WHERE id=?"; + public static String selectAllBucketQuery = "SELECT name, id, diskPoolId, versioning, MfaDelete, userName, userId, acl, web, cors, lifecycle, access, tagging, replication, encryption, objectlock, policy, createTime, replicaCount, usedSpace, fileCount, logging, objTagIndexing FROM BUCKETS"; public static String updateBucketQuery = "UPDATE BUCKETS SET versioning=? WHERE id=?"; public static String updateBucketAclQuery = "UPDATE BUCKETS SET acl=? WHERE id=?"; @@ -76,6 +76,7 @@ public final class DataRepositoryQuery { public static String updateBucketFilecountQuery = "UPDATE BUCKETS SET fileCount = fileCount + ? WHERE id=?"; public static String updateBucketUsedSpaceQuery = "UPDATE BUCKETS SET usedSpace = usedSpace + ? WHERE id=?"; public static String updateBucketLoggingQuery = "UPDATE BUCK SET logging=? WHERE id=?"; + public static String updateBucketObjTagIndexingQuery = "UPDATE BUCK SET objTagIndexing=? WHERE id=?"; // for multipart public static String createMultiPartQuery= "CREATE TABLE IF NOT EXISTS MULTIPARTS(" @@ -131,4 +132,30 @@ public final class DataRepositoryQuery { public static String selectLifeCycleQuery = "SELECT idx, bucket, objKey, objid, versionid, uploadid, inDate, log FROM %s WHERE objid=? AND AND versionid=?"; public static String selectAllLifeCycleQuery = "SELECT idx, bucket, objKey, versionid, uploadid, inDate, log FROM %s"; public static String deleteLifeCycleQuery = "DELETE FROM %s WHERE objKey=? AND versionid=?"; + + // for tags indexing + public static String tagIndexingTablePrefexi = "_ObjTagIndex"; + + public static String createTagIndexingQuery= "CREATE TABLE IF NOT EXISTS %s(" + + " objid VARCHAR(50) NOT NULL, " + + " versionid VARCHAR(50) NOT NULL DEFAULT 'nil'," + + " TagKey VARCHAR(191) NOT NULL," + + " TagValue VARCHAR(256) NOT NULL," + + " PRIMARY KEY(objid, versionid, TagKey)) ENGINE=INNODB DEFAULT CHARSET=UTF8mb4 COLLATE=utf8mb4_unicode_ci;"; + public static String insertTagIndexingQuery = "INSERT INTO %s(objid, versionid, TagKey, TagValue) VALUES(?, ?, ?, ?)"; + public static String deleteTagIndexingQuery1 = "DELETE FROM %s WHERE objid=? AND versionid=?"; + public static String deleteTagIndexingQuery2 = "DELETE FROM %s WHERE objid=? AND versionid=? AND TagKey=?"; + public static String selectTagIndexingQuery = "SELECT objid, versionid, TagKey, TagValue FROM %s WHERE %s"; + + // for restore objects + public static String createRestoreObjectsQuery= "CREATE TABLE IF NOT EXISTS RESTOREOBJECTS(" + + " bucket VARCHAR(256) NOT NULL DEFAULT '' COMMENT 'bucket name'," + + " objKey VARBINARY(2048) COMMENT 'Object key'," + + " objid VARCHAR(50) NOT NULL, " + + " versionid VARCHAR(50) NOT NULL DEFAULT 'nil'," + + " request TEXT," + + " PRIMARY KEY(objid, versionid)) ENGINE=INNODB DEFAULT CHARSET=UTF8mb4 COLLATE=utf8mb4_unicode_ci;"; + public static String insertRestoreObjectsQuery = "INSERT INTO RESTOREOBJECTS(bucket, objKey, objid, versionid, request) VALUES(?, ?, ?, ?, ?)"; + public static String selectRestoreObjectsQuery = "SELECT request FROM RESTOREOBJECTS WHERE objid=? AND versionid=?"; + public static String deleteRestoreObjectsQuery = "DELETE FROM RESTOREOBJECTS WHERE objid=? AND versionid=?"; } diff --git a/core/src/com/pspace/ifs/ksan/objmanager/DiskMonitor.java b/core/src/com/pspace/ifs/ksan/objmanager/DiskMonitor.java index 9074da2f..f83fff01 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/DiskMonitor.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/DiskMonitor.java @@ -298,6 +298,8 @@ private JsonOutput decodeJsonData(String msg)throws ParseException{ String status = (String)jO.get(KEYS.REPLICACOUNT.label); if (status.equalsIgnoreCase("OnePlusOne")) res.replicaCount = 2; + else if (status.equalsIgnoreCase("ErasureCode")) + res.replicaCount = 2; else res.replicaCount = 1; } diff --git a/core/src/com/pspace/ifs/ksan/objmanager/GetFromPortal.java b/core/src/com/pspace/ifs/ksan/objmanager/GetFromPortal.java index 7b9a0660..e1d6655d 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/GetFromPortal.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/GetFromPortal.java @@ -324,6 +324,8 @@ private void parseDiskPoolsResponse(ObjManagerCache omc, String response) throws DISKPOOL dp = new DISKPOOL(diskpoolId, diskpoolName); if (replication.equalsIgnoreCase("OnePlusOne")) dp.setDefaultReplicaCount(2); + else if (replication.equalsIgnoreCase("ErasureCode")) + dp.setDefaultReplicaCount(2); else dp.setDefaultReplicaCount(1); diff --git a/core/src/com/pspace/ifs/ksan/objmanager/MongoDataRepository.java b/core/src/com/pspace/ifs/ksan/objmanager/MongoDataRepository.java index 6f571657..a048d19a 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/MongoDataRepository.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/MongoDataRepository.java @@ -114,6 +114,7 @@ public class MongoDataRepository implements DataRepository{ private static final String FILECOUNT="fileCount"; private static final String USEDSPACE="usedSpace"; private static final String LOGGING="logging"; + private static final String OBJECTTAG_INDEXING = "objTagIndexing"; // for multipart upload private static final String UPLOADID="uploadId"; @@ -135,6 +136,15 @@ public class MongoDataRepository implements DataRepository{ private static final String LOGMSG = "log"; private static final String LIFECYCLEID= "idx"; private static final String ISFAILED = "isFailed"; + + // for Object Tag indexing + private static final String OBJTAGINDEX_TABLE = "ObjTagIndex"; + private static final String TAG_KEY = "TagKey"; + private static final String TAG_VALUE = "TagValue"; + + // for restore objects + private static final String OBJRESTORE_TABLE = "RESTOREOBJECTS"; + private static final String OBJRESTORE_REQUEST = "request"; public MongoDataRepository(ObjManagerCache obmCache, String hosts, String username, String passwd, String dbname, int port) throws UnknownHostException{ this.username = username; @@ -146,6 +156,7 @@ public MongoDataRepository(ObjManagerCache obmCache, String hosts, String usern createBucketsHolder(); createLifCycleHolder(LIFECYCLESEVENTS); createLifCycleHolder(LIFECYCLESFAILEDEVENTS); + createRestoreObjHolder(); } private void parseDBHostNames2URL(String hosts, int port){ @@ -211,6 +222,35 @@ private void createLifCycleHolder(String collectionName){ } } + private void createObjTagHolder(String bucketName){ + Document index; + String collectionName = bucketName + "_" + OBJTAGINDEX_TABLE; + //objid, versionid, uploadid + index = new Document(OBJID, 1); + index.append(VERSIONID, 1); + index.append(TAG_KEY, 1); + MongoCollection objTag = database.getCollection(collectionName); + if (objTag == null){ + database.createCollection(collectionName); + objTag = database.getCollection(collectionName); + objTag.createIndex(index, new IndexOptions().unique(true)); + } + } + + private MongoCollection getObjTagIndexCollection(String bucketName){ + MongoCollection objTag; + String collectionName = bucketName + "_" + OBJTAGINDEX_TABLE; + + objTag = this.database.getCollection(collectionName); + if (objTag == null){ + database.createCollection(collectionName); + objTag = database.getCollection(collectionName); + objTag.createIndex(Indexes.ascending(OBJID, VERSIONID, TAG_KEY), new IndexOptions().unique(true)); + } + + return objTag; + } + private MongoCollection getLifCyclesCollection(String collectionName){ MongoCollection lifeCycle; @@ -237,6 +277,31 @@ private MongoCollection getMultiPartUploadCollection(){ return multip; } + private void createRestoreObjHolder(){ + Document index; + + index = new Document(OBJID, 1); + index.append(VERSIONID, 1); + //index.append(BUCKETNAME, 1); + MongoCollection restoreObj = database.getCollection(OBJRESTORE_TABLE); + if (restoreObj == null){ + database.createCollection(OBJRESTORE_TABLE); + restoreObj = database.getCollection(OBJRESTORE_TABLE); + restoreObj.createIndex(index, new IndexOptions().unique(true)); + } + } + + private MongoCollection getRestoreObjCollection(){ + MongoCollection restoreObj; + + restoreObj = this.database.getCollection(OBJRESTORE_TABLE); + if (restoreObj == null){ + database.createCollection(OBJRESTORE_TABLE); + restoreObj = database.getCollection(OBJRESTORE_TABLE); + restoreObj.createIndex(Indexes.ascending(OBJID, VERSIONID), new IndexOptions().unique(true)); + } + return restoreObj; + } private String getCurrentDateTime(){ SimpleDateFormat formatter = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss"); @@ -254,6 +319,23 @@ private Date convertString2Date(String dateStr){ return date; } + private String convertXML2Json(String xmlstr){ + String tags[] = xmlstr.split(""); + if (tags.length > 1){ + String tag = tags[1].replaceAll("", "").replaceAll("", ""); + tag = tag.replaceAll("", "{"); + tag = tag.replaceAll("", "}"); + tag = tag.replaceAll("", ""); + tag = tag.replaceAll("", ","); + tag = tag.replaceAll("", "\""); + tag = tag.replaceAll("", "\": "); + tag = tag.replaceAll("", " \""); + tag = tag.replaceAll("", "\" "); + return tag; + } + return null; + } + @Override public int insertObject(Metadata md) throws ResourceNotFoundException { MongoCollection objects; @@ -279,6 +361,8 @@ public int insertObject(Metadata md) throws ResourceNotFoundException { objects.updateMany(Filters.eq(OBJID, md.getObjId()), Updates.set(LASTVERSION, false)); objects.insertOne(doc); updateBucketObjectCount(md.getBucket(), 1); + + insertObjTag(md.getBucket(), md.getObjId(), md.getVersionId(), md.getTag()); return 0; } @@ -429,50 +513,48 @@ public List getAllUsedDiskId() throws SQLException{ public Bucket insertBucket(Bucket bt) throws ResourceAlreadyExistException{ Document doc; Document index; - //try{ - doc = new Document(BUCKETNAME, bt.getName()); - //doc.append(BUCKETNAME, bucketName); - doc.append(DISKPOOLID, bt.getDiskPoolId()); - doc.append(BUCKETID, new Metadata(bt.getName(), "/").getBucketId()); - doc.append(USERID, bt.getUserId()); - doc.append(ACL, bt.getAcl()); - doc.append(REPLICACOUNT, bt.getReplicaCount()); - - doc.append(USERNAME, bt.getUserName()); - doc.append(WEB, ""); - doc.append(CORS, ""); - doc.append(LIFECYCLE, ""); - doc.append(ACCESS, ""); - doc.append(TAGGING, ""); - doc.append(REPLICATION, ""); - doc.append(VERSIONING, ""); - doc.append(MFADELETE, ""); - - doc.append(ENCRYPTION, bt.getEncryption()); - doc.append(OBJECTLOCK, bt.getObjectLock()); - doc.append(POLICY, ""); - doc.append(FILECOUNT, 0L); - doc.append(USEDSPACE, 0L); - doc.append(CREATETIME, getCurrentDateTime()); - doc.append(LOGGING, ""); - - buckets.insertOne(doc); - database.createCollection(bt.getName()); - // for index for object collection - index = new Document(OBJID, 1); - index.append(VERSIONID, 1); - index.append(LASTVERSION, 1); - index.append(DELETEMARKER, 1); - //index.append(OBJKEY, 1); - database.getCollection(bt.getName()).createIndex(index, new IndexOptions().unique(true)); - // wild index for listobjects - Document wildIndex = new Document(OBJID + ".$**", 1); - database.getCollection(bt.getName()).createIndex(wildIndex); - //bt = new Bucket(bucketName, bucketName, diskPoolId); - //getUserDiskPool(bt); - /*} catch(SQLException ex){ - throw new ResourceAlreadyExistException(String.format("Bucket(%s) is laready exist in the db!", bt.getName()), ex); - }*/ + + doc = new Document(BUCKETNAME, bt.getName()); + //doc.append(BUCKETNAME, bucketName); + doc.append(DISKPOOLID, bt.getDiskPoolId()); + doc.append(BUCKETID, new Metadata(bt.getName(), "/").getBucketId()); + doc.append(USERID, bt.getUserId()); + doc.append(ACL, bt.getAcl()); + doc.append(REPLICACOUNT, bt.getReplicaCount()); + + doc.append(USERNAME, bt.getUserName()); + doc.append(WEB, ""); + doc.append(CORS, ""); + doc.append(LIFECYCLE, ""); + doc.append(ACCESS, ""); + doc.append(TAGGING, bt.getTagging()); + doc.append(REPLICATION, ""); + doc.append(VERSIONING, ""); + doc.append(MFADELETE, ""); + + doc.append(ENCRYPTION, bt.getEncryption()); + doc.append(OBJECTLOCK, bt.getObjectLock()); + doc.append(POLICY, ""); + doc.append(FILECOUNT, 0L); + doc.append(USEDSPACE, 0L); + doc.append(CREATETIME, getCurrentDateTime()); + doc.append(LOGGING, bt.getLogging()); + doc.append(OBJECTTAG_INDEXING, bt.isObjectTagIndexEnabled()); + + buckets.insertOne(doc); + database.createCollection(bt.getName()); + // for index for object collection + index = new Document(OBJID, 1); + index.append(VERSIONID, 1); + index.append(LASTVERSION, 1); + index.append(DELETEMARKER, 1); + //index.append(OBJKEY, 1); + database.getCollection(bt.getName()).createIndex(index, new IndexOptions().unique(true)); + // wild index for listobjects + Document wildIndex = new Document(OBJID + ".$**", 1); + database.getCollection(bt.getName()).createIndex(wildIndex); + + createObjTagHolder(bt.getName()); return bt; } @@ -480,6 +562,7 @@ public Bucket insertBucket(Bucket bt) throws ResourceAlreadyExistException{ public int deleteBucket(String bucketName) { buckets.deleteOne(Filters.eq(BUCKETNAME, bucketName)); this.database.getCollection(bucketName).drop(); + getObjTagIndexCollection(bucketName).drop(); return 0; } @@ -515,6 +598,9 @@ private Bucket parseBucket(String bucketName, Document doc) throws ResourceNotFo long usedSpace = getParseLong(doc, USEDSPACE); long fileCount = getParseLong(doc, FILECOUNT); String logging = doc.getString(LOGGING); + boolean objTagIndexing = false; + if (doc.containsKey(OBJECTTAG_INDEXING)) + objTagIndexing = doc.getBoolean(OBJECTTAG_INDEXING); Date createTime; try { @@ -549,6 +635,7 @@ private Bucket parseBucket(String bucketName, Document doc) throws ResourceNotFo bt.setFileCount(fileCount); bt.setUsedSpace(usedSpace); bt.setLogging(logging); + bt.setObjectTagIndexEnabled(objTagIndexing); return bt; } @@ -653,67 +740,13 @@ public int deleteMultipartUpload(String bucket, String uploadid) throws SQLExce @Override public List selectMultipart(String bucket, String uploadid, int maxParts, int partNoMarker) throws SQLException{ List list=new ArrayList<>(); - /*MongoCollection multip; - String collName; - - collName = getMultiPartUploadCollName(bucket); - multip = this.database.getCollection(collName); - if (multip == null) - return null; - - FindIterable fit = multip.find(Filters.and(eq(UPLOADID, uploadid), Filters.gt(PARTNO, partNoMarker))) - .limit(maxParts).sort(new BasicDBObject(PARTNO, 1 )); - - Iterator it = fit.iterator(); - while((it.hasNext())){ - Document doc = (Document)it.next(); - list.add(doc.getInteger(PARTNO)); - }*/ return list; } @Override public void selectMultipartUpload(String bucket, Object query, int maxKeys, DBCallBack callback) throws SQLException { - /*MongoCollection multip; - String collName; - String key; - String uploadid; - long partNo; - int counter = 0; - boolean isTruncated = false; - - collName = getMultiPartUploadCollName(bucket); - multip = this.database.getCollection(collName); - - BasicDBObject sortList = new BasicDBObject(OBJKEY, 1 ); - sortList.append(UPLOADID, 1); - sortList.append(PARTNO, 1); - FindIterable fit = multip.find((BasicDBObject)query).limit(maxKeys + 1) - .sort(sortList); - - Iterator it = fit.iterator(); - while((it.hasNext())){ - Document doc = (Document)it.next(); - key = doc.getString(OBJKEY); - uploadid = doc.getString(UPLOADID); - partNo = (long )doc.getInteger(PARTNO); - ++counter; - if (counter == maxKeys){ - isTruncated =it.hasNext(); - System.out.println("counter : "+ counter + " max :" + maxKeys + " next :" + isTruncated); - } - - callback.call(key, uploadid, "", partNo, "", "", "", isTruncated); - if (isTruncated == true) - break; - }*/ + } - - /*@Override - public ObjectListParameter selectObjects(String bucketName, Object query, int maxKeys) throws SQLException { - // TODO Auto-generated method stub - return null; - }*/ private int updateVersionDelete(String bucketName, String objId){ MongoCollection objects; @@ -769,6 +802,8 @@ public int deleteObject(String bucketName, String objKey, String versionId) { updateVersionDelete(bucketName, objId); updateBucketObjectCount(bucketName, -1); + + removeObjTag(bucketName, objId, versionId, ""); } return nchange; } @@ -972,6 +1007,16 @@ private int updateBucket(String bucketName, String key, String value){ return 0; } + private int updateBucket(String bucketName, String key, boolean value){ + FindIterable fit = buckets.find(eq(BUCKETNAME, bucketName)); + Document doc =(Document)fit.first(); + if (doc == null) + return -1; + + buckets.updateOne(Filters.eq(BUCKETNAME, bucketName), Updates.set(key, value)); + return 0; + } + private int updateBucketObjectSpaceCount(String bucketName, String key, long value){ FindIterable fit = buckets.find(eq(BUCKETNAME, bucketName)); Document doc =(Document)fit.first(); @@ -1022,6 +1067,11 @@ public void updateBucketLogging(Bucket bt) throws SQLException { updateBucket(bt.getName(), LOGGING, bt.getLogging()); } + @Override + public void updateBucketObjTagIndexing(Bucket bt) throws SQLException { + updateBucket(bt.getName(), OBJECTTAG_INDEXING, bt.isObjectTagIndexEnabled()); + } + private List getUtilJobObject(String Id, String status, long TotalNumObject, long NumJobDone, boolean checkOnly, String utilName, String startTime){ List res = new ArrayList<>(); @@ -1230,6 +1280,7 @@ public long getObjectListCount(String bucketName, Object query) throws SQLExcept public void updateObjectTagging(Metadata mt) throws SQLException { updateObject(mt.getBucket(), mt.getObjId(), mt.getVersionId(), TAG, mt.getTag()); updateObject(mt.getBucket(), mt.getObjId(), mt.getVersionId(), META, mt.getMeta()); + insertObjTag(mt.getBucket(), mt.getObjId(), mt.getVersionId(), mt.getTag()); } @Override @@ -1436,4 +1487,169 @@ public int deleteLifeCycle(LifeCycle lc) throws SQLException { public int deleteFailedLifeCycle(LifeCycle lc) throws SQLException { return deleteLifeCycle(LIFECYCLESFAILEDEVENTS, lc); } + + private int insertObjTag(String bucketName, String objId, String versionId, String tags) { + Bucket bt; + MongoCollection objTags = getObjTagIndexCollection(bucketName); + + if (tags == null) + return 0; + + if (tags.isEmpty()) + return 0; + + try { + bt = selectBucket(bucketName); + if (!bt.isObjectTagIndexEnabled()) + return 0; + } catch (ResourceNotFoundException | SQLException ex) { + return 0; + } + + if (objTags == null){ + //throw new SQLException("[insertObjTag] mongo db holder for object Tags indexing not found!"); + return 0; + } + + String tagsJson = convertXML2Json(tags); + if (tagsJson == null) + return 0; // ignore + + if (tagsJson.isEmpty()) + return 0; // ignore + + Document doc = new Document(OBJID, objId); + doc.append(VERSIONID, versionId); + + + Document tagDoc = Document.parse(tagsJson); + String key, value; + Iterator it = tagDoc.keySet().iterator(); + while(it.hasNext()){ + key = (String)it.next(); + value = tagDoc.getString(key); + if (doc.containsKey(TAG_KEY)) + doc.replace(TAG_KEY, key); + else + doc.append(TAG_KEY, key); + if (doc.containsKey(TAG_VALUE)) + doc.replace(TAG_VALUE, value); + else + doc.append(TAG_VALUE, value); + + objTags.insertOne(doc); + } + return 0; + } + + private int removeObjTag(String bucketName, String objId, String versionId, String tags){ + MongoCollection objTags = getObjTagIndexCollection(bucketName); + int ret = 0; + + if (objTags == null){ + return 0; + //throw new SQLException("[removeObjTag] mongo db holder for object Tags indexing not found!"); + } + + String tagsJson = convertXML2Json(tags); + if (tagsJson == null){ + objTags.findOneAndDelete(Filters.and(eq(OBJID, objId), eq(VERSIONID, versionId))); + ret++; + return ret; + } + + if (tagsJson.isEmpty()){ + objTags.findOneAndDelete(Filters.and(eq(OBJID, objId), eq(VERSIONID, versionId))); + ret++; + return ret; + } + + Document doc = Document.parse(tagsJson); + String key; + Iterator it = doc.keySet().iterator(); + if (it.hasNext()){ + while(it.hasNext()){ + key = (String)it.next(); + objTags.findOneAndDelete(Filters.and(eq(OBJID, objId), eq(VERSIONID, versionId), eq(TAG_KEY, key))); + ret++; + } + } + else{ + objTags.findOneAndDelete(Filters.and(eq(OBJID, objId), eq(VERSIONID, versionId))); + ret++; + } + return ret; + } + + @Override + public List listObjectWithTags(String bucketName, Object query, int maxObjects) throws SQLException{ + MongoCollection objectsTags; + objectsTags = getObjTagIndexCollection(bucketName); + BasicDBObject mongoQuery =(BasicDBObject)query; + + FindIterable oit = objectsTags.find(mongoQuery).limit(maxObjects); + Iterator it = oit.iterator(); + List list = new ArrayList(); + while((it.hasNext())){ + Document doc = (Document)it.next(); + String objId = doc.getString(OBJID); + String versionid = doc.getString(VERSIONID); + Metadata mt; + try { + mt = selectSingleObjectInternal(bucketName, objId, versionid); + list.add(mt); + } catch (ResourceNotFoundException ex) { + //Logger.getLogger(MongoDataRepository.class.getName()).log(Level.SEVERE, null, ex); + } + } + return list; + } + + @Override + public int insertRestoreObjectRequest(String bucketName, String key, String objId, String versionId, String request) throws SQLException{ + MongoCollection objRestore = getRestoreObjCollection(); + + if (request == null) + return -1; + + if (request.isEmpty()) + return -1; + + if (objRestore == null){ + //throw new SQLException("[insertRestoreObjectRequest] mongo db holder for restore objects not found!"); + return -1; + } + + Document doc = new Document(BUCKETNAME, bucketName); + doc.append(OBJID, objId); + doc.append(OBJKEY, key); + doc.append(VERSIONID, versionId); + doc.append(OBJRESTORE_REQUEST, request); + objRestore.insertOne(doc); + + return 0; + } + + @Override + public String getRestoreObjectRequest(String bucketName, String objId, String versionId) throws SQLException{ + MongoCollection objRestore; + String request=null; + + objRestore = getRestoreObjCollection(); + FindIterable oit = objRestore.find(Filters.and(eq(OBJID, objId), eq(VERSIONID, versionId))); + Iterator it = oit.iterator(); + if((it.hasNext())){ + Document doc = (Document)it.next(); + request = doc.getString(OBJRESTORE_REQUEST); + } + return request; + } + + @Override + public void deleteRestoreObjectRequest(String bucketName, String objId, String versionId) throws SQLException{ + MongoCollection objRestore; + + objRestore = getRestoreObjCollection(); + objRestore.findOneAndDelete(Filters.and(eq(OBJID, objId), eq(VERSIONID, versionId))); + } } diff --git a/core/src/com/pspace/ifs/ksan/objmanager/MysqlDataRepository.java b/core/src/com/pspace/ifs/ksan/objmanager/MysqlDataRepository.java index 5bd763f6..06e70f93 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/MysqlDataRepository.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/MysqlDataRepository.java @@ -24,11 +24,13 @@ import java.time.LocalDateTime; import java.time.format.DateTimeFormatter; import java.util.ArrayList; +import java.util.Iterator; import java.util.List; import java.util.SortedMap; import java.util.TreeMap; import java.util.logging.Level; import java.util.logging.Logger; +import org.bson.Document; /** * * @author legesse @@ -64,6 +66,7 @@ public class MysqlDataRepository implements DataRepository{ private PreparedStatement pstUpdateBucketPolicy; private PreparedStatement pstUpdateBucketFilecount; private PreparedStatement pstUpdateBucketUsedSpace; + private PreparedStatement pstUpdateObjTagIndexBucket; // for multipart upload private PreparedStatement pstCreateMultiPart; @@ -86,13 +89,11 @@ public class MysqlDataRepository implements DataRepository{ private PreparedStatement pstSelectUJob; - // for LifeCycle - //private PreparedStatement pstCreateLifeCycle; - // private PreparedStatement pstInsertLifeCycle; - //private PreparedStatement pstSelectLifeCycle; - //private PreparedStatement pstSelectByUploadIdLifeCycle; - //private PreparedStatement pstSelectAllLifeCycle; - //private PreparedStatement pstDeleteLifeCycle; + // for restore object + private PreparedStatement pstCreateRestoreObjects; + private PreparedStatement pstInsertRestoreObjects; + private PreparedStatement pstSelectRestoreObjects; + private PreparedStatement pstDeleteRestoreObjects; public MysqlDataRepository(ObjManagerCache obmCache, String host, String username, String passwd, String dbname) throws SQLException{ this.obmCache = obmCache; @@ -124,6 +125,7 @@ public MysqlDataRepository(ObjManagerCache obmCache, String host, String userna pstUpdateBucketFilecount = con.prepareStatement(DataRepositoryQuery.updateBucketFilecountQuery); pstUpdateBucketUsedSpace = con.prepareStatement(DataRepositoryQuery.updateBucketUsedSpaceQuery); //pstIsDeleteBucket = con.prepareStatement(DataRepositoryQuery.objIsDeleteBucketQuery); + pstUpdateObjTagIndexBucket = con.prepareStatement(DataRepositoryQuery.updateBucketObjTagIndexingQuery); // for multipart pstCreateMultiPart= con.prepareStatement(DataRepositoryQuery.createMultiPartQuery); @@ -148,12 +150,10 @@ public MysqlDataRepository(ObjManagerCache obmCache, String host, String userna pstSelectUJob = con.prepareStatement(DataRepositoryQuery.selectUJobQuery); // for LifeCycle - //pstCreateLifeCycle = con.prepareStatement(DataRepositoryQuery.createLifeCycleQuery); - //pstInsertLifeCycle = con.prepareStatement(DataRepositoryQuery.insertLifeCycleQuery); - //pstSelectLifeCycle = con.prepareStatement(DataRepositoryQuery.selectLifeCycleQuery); - //pstSelectByUploadIdLifeCycle = con.prepareStatement(DataRepositoryQuery.selectByUploadIdLifeCycleQuery); - //pstSelectAllLifeCycle = con.prepareStatement(DataRepositoryQuery.selectAllLifeCycleQuery); - //pstDeleteLifeCycle = con.prepareStatement(DataRepositoryQuery.deleteLifeCycleQuery); + pstCreateRestoreObjects = con.prepareStatement(DataRepositoryQuery.createRestoreObjectsQuery); + pstInsertRestoreObjects = con.prepareStatement(DataRepositoryQuery.insertRestoreObjectsQuery); + pstSelectRestoreObjects = con.prepareStatement(DataRepositoryQuery.selectRestoreObjectsQuery); + pstDeleteRestoreObjects = con.prepareStatement(DataRepositoryQuery.deleteRestoreObjectsQuery); } catch(SQLException ex){ this.ex_message(ex); @@ -210,6 +210,7 @@ private int createTable() throws SQLException{ pstCreateUJob.execute(); //pstCreateLifeCycle.execute(); createLifeCycleEventTables(); + createRestoreObjectTable(); return 0; } @@ -236,6 +237,23 @@ private PreparedStatement getObjPreparedStmt(String bucketName, String format) t return pstStmt; }*/ + private String convertXML2Json(String xmlstr){ + String tags[] = xmlstr.split(""); + if (tags.length > 1){ + String tag = tags[1].replaceAll("", "").replaceAll("", ""); + tag = tag.replaceAll("", "{"); + tag = tag.replaceAll("", "}"); + tag = tag.replaceAll("", ""); + tag = tag.replaceAll("", ","); + tag = tag.replaceAll("", "\""); + tag = tag.replaceAll("", "\": "); + tag = tag.replaceAll("", " \""); + tag = tag.replaceAll("", "\" "); + return tag; + } + return null; + } + private void createObjectTable(String bucketName) throws SQLException{ PreparedStatement pstStmt = getObjPreparedStmt(bucketName, DataRepositoryQuery.objCreateQuery); pstStmt.execute(); @@ -589,6 +607,7 @@ public void updateObjectTagging(Metadata mt) throws SQLException { pstUpdateTagging.setString(3, mt.getObjId()); pstUpdateTagging.setString(4, mt.getVersionId()); pstUpdateTagging.executeUpdate(); + insertObjTag(mt.getBucket(), mt.getObjId(), mt.getVersionId(), mt.getTag()); } @Override @@ -634,6 +653,7 @@ public int deleteObject(String bucketName, String path, String versionId) { if (!versionId.isEmpty()) updateVersionDelete(bucketName, objId); } + removeObjTag(bucketName, objId, versionId); } catch(SQLException ex){ this.ex_message(ex); return -ex.getErrorCode(); @@ -694,11 +714,18 @@ public synchronized Bucket insertBucket(Bucket bt) this.pstInsertBucket.setString(7, bt.getEncryption()); this.pstInsertBucket.setString(8, bt.getObjectLock()); this.pstInsertBucket.setInt(9, bt.getReplicaCount()); + this.pstInsertBucket.setBoolean(10, bt.isObjectTagIndexEnabled()); this.pstInsertBucket.executeUpdate(); - //getUserDiskPool(bt); // get diskpoolId and replicaCount } catch(SQLException ex){ System.out.println("SQLException:>" + ex); - throw new ResourceAlreadyExistException(String.format("Bucket(%s) is laready exist in the db!\n", bt.getName()), ex); + String str = String.format("name : %s, id : %s, diskPoolId : %s, userName : %s userId : %s, acl : %s, encryption : %s, objectlock : %s, replicaCount : %s, objTagIndexing : %s" + , bt.getName(), bt.getId(), bt.getDiskPoolId(), bt.getUserName(), bt.getUserId(), bt.getAcl(), bt.getEncryption(), bt.getObjectLock(), bt.getReplicaCount(), bt.isObjectTagIndexEnabled()); + throw new ResourceAlreadyExistException(String.format("Bucket(%s) is laready exist in the db!\n", str, ex)); + } + try { + createObjectTagIndexingTable(bt.getName()); + } catch (SQLException ex) { + Logger.getLogger(MysqlDataRepository.class.getName()).log(Level.SEVERE, null, ex); } return bt; } @@ -746,6 +773,7 @@ private Bucket parseBucket(ResultSet rs) throws SQLException{ long usedSpace = rs.getLong(20); long fileCount = rs.getLong(21); String logging = rs.getString(22); + boolean isObjTagIndexing = rs.getBoolean(23); Bucket bt = new Bucket(name, id, diskPoolId, versioning, mfaDelete, userId, acl, createTime); bt.setUserName(userName); @@ -763,6 +791,7 @@ private Bucket parseBucket(ResultSet rs) throws SQLException{ bt.setUsedSpace(usedSpace); bt.setFileCount(fileCount); bt.setLogging(logging); + bt.setObjectTagIndexEnabled(isObjTagIndexing); return bt; } @@ -835,6 +864,13 @@ public synchronized int updateBucketVersioning(Bucket bt) { } return -1; } + @Override + public synchronized void updateBucketObjTagIndexing(Bucket bt) throws SQLException { + pstUpdateObjTagIndexBucket.clearParameters(); + pstUpdateObjTagIndexBucket.setBoolean(1, bt.isObjectTagIndexEnabled()); + pstUpdateObjTagIndexBucket.setString(2, getBucketId(bt.getName())); + pstUpdateObjTagIndexBucket.executeUpdate(); + } @Override public synchronized int insertMultipartUpload(String bucket, String objkey, String uploadid, int partNo, String acl, String meta, String etag, long size, String pdiskid) throws SQLException{ @@ -1419,4 +1455,164 @@ public int deleteLifeCycle(LifeCycle lc) throws SQLException{ public int deleteFailedLifeCycle(LifeCycle lc) throws SQLException{ return deleteLifeCycle(DataRepositoryQuery.lifeCycleFailedEventTableName, lc); } + + private void createObjectTagIndexingTable(String bucketName) throws SQLException{ + PreparedStatement pstStmt = getObjPreparedStmt(bucketName + DataRepositoryQuery.tagIndexingTablePrefexi, DataRepositoryQuery.createTagIndexingQuery); + pstStmt.execute(); + } + + private int insertObjTag(String bucketName, String objId, String versionId, String tags) { + + try { + String tkey; + String tvalue; + Bucket bt; + int ret = 0; + + String tagsJson = convertXML2Json(tags); + if (tagsJson == null) + return 0; // ignore + + if (tagsJson.isEmpty()) + return 0; // ignore + + try { + bt = selectBucket(bucketName); + if (!bt.isObjectTagIndexEnabled()) + return 0; + } catch (ResourceNotFoundException | SQLException ex) { + return 0; + } + + //removeObjTag(bucketName, objId, versionId, tags); + PreparedStatement pstInsertStmt = getObjPreparedStmt(bucketName + DataRepositoryQuery.tagIndexingTablePrefexi, DataRepositoryQuery.insertTagIndexingQuery); + + pstInsertStmt.clearParameters(); + pstInsertStmt.setString(1, objId); + pstInsertStmt.setString(2, versionId); + Document tagDoc = Document.parse(tagsJson); + Iterator it = tagDoc.keySet().iterator(); + while(it.hasNext()){ + tkey = (String)it.next(); + tvalue = tagDoc.getString(tkey); + pstInsertStmt.setString(3, tkey); + pstInsertStmt.setString(4, tvalue); + if (pstInsertStmt.execute()) + ret++; + } + return ret; + } catch (SQLException ex) { + Logger.getLogger(MysqlDataRepository.class.getName()).log(Level.SEVERE, null, ex); + return 0; + } + } + + private int removeObjTag(String bucketName, String objId, String versionId) throws SQLException{ + PreparedStatement pstdeleteStmt = getObjPreparedStmt(bucketName + DataRepositoryQuery.tagIndexingTablePrefexi, DataRepositoryQuery.deleteTagIndexingQuery1); + pstdeleteStmt.clearParameters(); + pstdeleteStmt.setString(1, objId); + pstdeleteStmt.setString(2, versionId); + return pstdeleteStmt.executeUpdate(); + } + + private int removeObjTag(String bucketName, String objId, String versionId, String tags) throws SQLException{ + String key; + String tagsJson = convertXML2Json(tags); + if (tagsJson == null) + return 0; + + if (tagsJson.isEmpty()) + return 0; + + PreparedStatement pstdeleteStmt = getObjPreparedStmt(bucketName + DataRepositoryQuery.tagIndexingTablePrefexi, DataRepositoryQuery.deleteTagIndexingQuery2); + pstdeleteStmt.clearParameters(); + pstdeleteStmt.setString(1, objId); + pstdeleteStmt.setString(2, versionId); + Document doc = Document.parse(tagsJson); + Iterator it = doc.keySet().iterator(); + if (it.hasNext()){ + while(it.hasNext()){ + key = (String)it.next(); + pstdeleteStmt.setString(3, key); + } + } + return pstdeleteStmt.executeUpdate(); + } + + private List parseSelectListObjectwithTags(String bucketName, ResultSet rs) throws SQLException{ + List list = new ArrayList(); + while(rs.next()){ + String objid = rs.getString("objid"); + String versionId = rs.getString("versionid"); + try { + list.add(this.selectSingleObjectInternal(bucketName, objid, versionId)); + } catch (ResourceNotFoundException ex) { + // skipe + } + } + return list; + } + + @Override + public List listObjectWithTags(String bucketName, Object query, int maxKeys) throws SQLException{ + PreparedStatement pstselectStmt = getObjPreparedStmt(bucketName+ DataRepositoryQuery.tagIndexingTablePrefexi, DataRepositoryQuery.selectTagIndexingQuery + (String)query); + ResultSet rs = pstselectStmt.executeQuery(); + return parseSelectListObjectwithTags(bucketName, rs); + } + + private void createRestoreObjectTable() throws SQLException{ + pstCreateRestoreObjects.execute(); + } + + private int insertRestoreObject(String bucketName, String objKey, String objId, String versionId, String request) throws SQLException { + + if (request == null) + return -1; // ignore + + if (request.isEmpty()) + return -1; // ignore + + pstInsertRestoreObjects.clearParameters(); + pstInsertRestoreObjects.setString(1, bucketName); + pstInsertRestoreObjects.setString(2, objKey); + pstInsertRestoreObjects.setString(3, objId); + pstInsertRestoreObjects.setString(4, versionId); + pstInsertRestoreObjects.setString(5, request); + pstInsertRestoreObjects.execute(); + return 0; + } + + private int removeRestoreObject(String bucketName, String objId, String versionId) throws SQLException{ + pstDeleteRestoreObjects.clearParameters(); + pstDeleteRestoreObjects.setString(1, objId); + pstDeleteRestoreObjects.setString(2, versionId); + return pstDeleteRestoreObjects.executeUpdate(); + } + + private String selectRestoreObject(String bucketName, String objId, String versionId) throws SQLException{ + String request = null; + pstSelectRestoreObjects.setString(1, objId); + pstSelectRestoreObjects.setString(2, versionId); + ResultSet rs = pstSelectRestoreObjects.executeQuery(); + if (rs.next()){ + request = rs.getString("request"); + } + + return request; + } + + @Override + public int insertRestoreObjectRequest(String bucketName, String key, String objId, String versionId, String request) throws SQLException { + return insertRestoreObject(bucketName, key, objId, versionId, request); + } + + @Override + public String getRestoreObjectRequest(String bucketName, String objId, String versionId) throws SQLException{ + return selectRestoreObject(bucketName, objId, versionId); + } + + @Override + public void deleteRestoreObjectRequest(String bucketName, String objId, String versionId) throws SQLException{ + removeRestoreObject(bucketName, objId, versionId); + } } diff --git a/core/src/com/pspace/ifs/ksan/objmanager/ObjManager.java b/core/src/com/pspace/ifs/ksan/objmanager/ObjManager.java index 8c27cb9d..e5be2857 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/ObjManager.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/ObjManager.java @@ -34,11 +34,14 @@ public class ObjManager { private DiskAllocation dAlloc; private static ObjManagerCache obmCache; private ObjManagerConfig config; - private OSDClient osdc; + //private OSDClient osdc; private ObjManagerSharedResource obmsr; private ObjMultipart multipart; private BucketManager bucketMGT; private Objects objectMGT; + private ObjTagsIndexing objectIndexing; + private RestoreObjects restoreObj; + private static Logger logger; private void init(ObjManagerConfig config) throws Exception { @@ -56,13 +59,17 @@ private void init(ObjManagerConfig config) throws Exception { dAlloc = new DiskAllocation(obmCache); - osdc = new OSDClient(config); + //osdc = new OSDClient(config); multipart = new ObjMultipart(dbm); bucketMGT = new BucketManager(dbm, obmCache); objectMGT = new Objects(dbm, dAlloc, obmCache, bucketMGT); + + objectIndexing = new ObjTagsIndexing(bucketMGT, objectMGT); + + restoreObj = new RestoreObjects(dbm); } public ObjManager() throws Exception { @@ -363,7 +370,7 @@ public void updateBucketObjectLock(String bucketName, String lock) throws SQLExc } public void updateBucketPolicy(String bucketName, String policy) throws SQLException, ResourceNotFoundException { - bucketMGT.updateBucketObjectLock(bucketName, policy); + bucketMGT.updateBucketPolicy(bucketName, policy); } public void updateBucketUsed(String bucketName, long size) throws SQLException, ResourceNotFoundException { @@ -387,6 +394,14 @@ public void updateDiskpools(String routingKey, String body){ this.obmsr.getDiskMonitor().update(routingKey, body); } + public ObjTagsIndexing getObjectTagsIndexing(){ + return this.objectIndexing; + } + + public RestoreObjects getRestoreObjects(){ + return this.restoreObj; + } + // for pool public boolean isValid(){ return true;} diff --git a/core/src/com/pspace/ifs/ksan/objmanager/ObjManagerUtil.java b/core/src/com/pspace/ifs/ksan/objmanager/ObjManagerUtil.java index 24487b79..495b8a62 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/ObjManagerUtil.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/ObjManagerUtil.java @@ -32,6 +32,7 @@ public class ObjManagerUtil { private ObjManagerSharedResource omsr; private LifeCycleManagment lfm; private ObjMultipart multipart; + private RestoreObjects restoreObj; private static Logger logger; public ObjManagerUtil(ObjManagerConfig config) throws Exception{ @@ -53,7 +54,9 @@ public ObjManagerUtil(ObjManagerConfig config) throws Exception{ lfm = new LifeCycleManagment(dbm); multipart = new ObjMultipart(dbm); - + + restoreObj = new RestoreObjects(dbm); + logger = LoggerFactory.getLogger(ObjManagerUtil.class); } @@ -61,7 +64,7 @@ public ObjManagerUtil() throws Exception{ this(new ObjManagerConfig()); } - private Bucket getBucket(String bucketName) throws ResourceNotFoundException, SQLException { + public Bucket getBucket(String bucketName) throws ResourceNotFoundException, SQLException { Bucket bt = obmCache.getBucketFromCache(bucketName); if (bt == null) bt = dbm.selectBucket(bucketName); @@ -260,4 +263,8 @@ public ObjMultipart getMultipartInsatance(String Bucket){ multipart.setBucket(Bucket); return multipart; } + + public RestoreObjects getRestoreObjects(){ + return this.restoreObj; + } } diff --git a/core/src/com/pspace/ifs/ksan/objmanager/ObjMultipart.java b/core/src/com/pspace/ifs/ksan/objmanager/ObjMultipart.java index e3881f47..bd7683e3 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/ObjMultipart.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/ObjMultipart.java @@ -181,7 +181,7 @@ public List listUploads(String delimiter, String prefix, String keyM DBCallBack cb = new ListMultipartCallBack(); //System.out.println("sql : " + sql); - dbm.selectMultipartUpload(bucket, sql, maxUploads, cb); + //dbm.selectMultipartUpload(bucket, sql, maxUploads, cb); return this.list; } diff --git a/core/src/com/pspace/ifs/ksan/objmanager/ObjTagsIndexing.java b/core/src/com/pspace/ifs/ksan/objmanager/ObjTagsIndexing.java new file mode 100644 index 00000000..da049457 --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/objmanager/ObjTagsIndexing.java @@ -0,0 +1,43 @@ +package com.pspace.ifs.ksan.objmanager; + +import com.pspace.ifs.ksan.objmanager.ObjManagerException.ResourceNotFoundException; +import java.sql.SQLException; +import java.util.HashMap; +import java.util.List; + +/* + * To change this license header, choose License Headers in Project Properties. + * To change this template file, choose Tools | Templates + * and open the template in the editor. + */ + +/** + * + * @author legesse + */ +public class ObjTagsIndexing { + private DataRepository dbm; + private final Objects objectMGT; + private final BucketManager bucketMGT; + + public ObjTagsIndexing(BucketManager bucketMGT, Objects objectMGT){ + this.bucketMGT = bucketMGT; + this.objectMGT = objectMGT; + } + + public void enableIndexing(String bucketName) throws SQLException, ResourceNotFoundException{ + bucketMGT.updateBucketTagsIndexing(bucketName, true); + } + + public void disableIndexing(String bucketName) throws SQLException, ResourceNotFoundException{ + bucketMGT.updateBucketTagsIndexing(bucketName, false); + } + + public boolean isIndexingEnabled(String bucketName) throws ResourceNotFoundException, SQLException{ + return bucketMGT.getBucket(bucketName).isObjectTagIndexEnabled(); + } + + public List getObjectWithTags(String bucketName, String tagList, int maxObjects) throws SQLException{ + return objectMGT.listObjectWithTags(bucketName, tagList, maxObjects); + } +} diff --git a/core/src/com/pspace/ifs/ksan/objmanager/Objects.java b/core/src/com/pspace/ifs/ksan/objmanager/Objects.java index d72bfa36..c633864f 100644 --- a/core/src/com/pspace/ifs/ksan/objmanager/Objects.java +++ b/core/src/com/pspace/ifs/ksan/objmanager/Objects.java @@ -12,6 +12,7 @@ package com.pspace.ifs.ksan.objmanager; +import com.mongodb.BasicDBObject; import com.pspace.ifs.ksan.libs.identity.ObjectListParameter; import com.pspace.ifs.ksan.libs.identity.S3ObjectList; import com.pspace.ifs.ksan.objmanager.ObjManagerException.AllServiceOfflineException; @@ -19,6 +20,9 @@ import java.io.IOException; import java.security.InvalidParameterException; import java.sql.SQLException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -172,6 +176,9 @@ private int _close(String bucketName, String key, Metadata mt) throws ResourceNo logger.debug("[OVERWRITE OBJECT] {}", mt2); dbm.deleteObject(bucketName, key, mt.getVersionId()); } + else if (mt.getVersionId().equalsIgnoreCase("null")){ + dbm.deleteObject(bucketName, key, mt.getVersionId()); + } } catch(ResourceNotFoundException ex){ } @@ -188,6 +195,66 @@ private int _close(String bucketName, String key, Metadata mt) throws ResourceNo return dbm.insertObject(mt); } + private HashMap getTagsKeyValue(String tagsList){ + String arr[]; + HashMap tags = new HashMap<>(); + + if (tagsList.isEmpty()) + return tags; + + if (tagsList.contains(",")) + arr = tagsList.split(","); + else{ + arr = new String[1]; + arr[0]= tagsList; + } + + for (String tag : arr){ + if (tag.contains(":")){ + String kv[] = tag.split(":"); + tags.put(kv[0], kv[1]); + } + else + tags.put(tag, " "); + } + return tags; + } + + private Object getListWithTagQuery(String tagsList){ + String sql = ""; + List and = new ArrayList(); + HashMap pair = getTagsKeyValue(tagsList); + if (pair.isEmpty()) + return null; + if (dbm instanceof MongoDataRepository){ + for (String key : pair.keySet()){ + and.add(new BasicDBObject("TagKey", new BasicDBObject("$eq", key))); + String value = pair.get(key); + if (!value.isEmpty()) + and.add(new BasicDBObject("TagValue", new BasicDBObject("$eq", value))); + } + + if (and.size() == 1) + return and.get(0); + else + return new BasicDBObject("$and", and.toArray()); + } + else { + for (String key : pair.keySet()){ + String value = pair.get(key); + if (sql.isEmpty()) + sql = sql + "WHERE TagKey=" + key; + else{ + sql = sql + " AND TagKey=" + key; + } + + if (!value.isEmpty()) + sql = sql + " AND TagValue=" + value; + } + return sql; + } + } + /******************************************************************/ public Metadata open(String bucketName, String key, String versionId) throws ResourceNotFoundException{ @@ -225,7 +292,8 @@ public void updateObjectTagging(Metadata mt) throws SQLException { public void updateObjectAcl(Metadata mt) throws SQLException { dbm.updateObjectAcl(mt); } - private static int parseMaxKeys(String maxKeysStr){ + + private static int parseMaxKeys(String maxKeysStr){ int maxKeys; try{ maxKeys = Integer.parseInt(maxKeysStr); @@ -249,4 +317,10 @@ public ObjectListParameter listObjectVersions(String bucketName, S3ObjectList s3 ListObject list = new ListObject(dbm, bucketName, s3ObjectList.getDelimiter(), s3ObjectList.getKeyMarker(), s3ObjectList.getVersionIdMarker(), parseMaxKeys(s3ObjectList.getMaxKeys()), s3ObjectList.getPrefix(), true); return list.getList(); } + + public List listObjectWithTags(String bucketName, String tagsList, int maxObjects) throws SQLException{ + Object query = getListWithTagQuery(tagsList); + logger.debug("[listObjectWithTags] tagsList> {} query> {} maxObjects {}", tagsList, query.toString(), maxObjects); + return dbm.listObjectWithTags(bucketName, query, maxObjects); + } } diff --git a/core/src/com/pspace/ifs/ksan/objmanager/RestoreObjects.java b/core/src/com/pspace/ifs/ksan/objmanager/RestoreObjects.java new file mode 100644 index 00000000..c4dbe1ee --- /dev/null +++ b/core/src/com/pspace/ifs/ksan/objmanager/RestoreObjects.java @@ -0,0 +1,44 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License. See LICENSE for details +* +* All materials such as this program, related source codes, and documents are provided as they are. +* Developers and developers of the KSAN project are not responsible for the results of using this program. +* The KSAN development team has the right to change the LICENSE method for all outcomes related to KSAN development without prior notice, permission, or consent. +*/ + +package com.pspace.ifs.ksan.objmanager; +import java.sql.SQLException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * + * @author legesse + */ +public class RestoreObjects { + private final DataRepository dbm; + private static Logger logger; + + public RestoreObjects(DataRepository dbm){ + logger = LoggerFactory.getLogger(Objects.class); + this.dbm = dbm; + } + + public int insertRequest(String bucketName, String key, String versionId, String request) throws SQLException{ + Metadata mt = new Metadata(bucketName, key); + return dbm.insertRestoreObjectRequest(bucketName, key, mt.getObjId(), versionId, request); + } + + public String getRequest(String bucketName, String key, String versionId) throws SQLException{ + Metadata mt = new Metadata(bucketName, key); + return dbm.getRestoreObjectRequest(bucketName, mt.getObjId(), versionId); + } + + public void removeRequest(String bucketName, String key, String versionId) throws SQLException{ + Metadata mt = new Metadata(bucketName, key); + dbm.deleteRestoreObjectRequest(bucketName, mt.getObjId(), versionId); + } +} diff --git a/core/src/com/pspace/ifs/ksan/osd/DoECPriObject.java b/core/src/com/pspace/ifs/ksan/osd/DoECPriObject.java index 92f89f87..acebde29 100644 --- a/core/src/com/pspace/ifs/ksan/osd/DoECPriObject.java +++ b/core/src/com/pspace/ifs/ksan/osd/DoECPriObject.java @@ -18,13 +18,23 @@ import java.util.Calendar; import java.util.HashMap; import java.util.List; +import java.io.FileInputStream; + +import org.apache.commons.io.FileUtils; import com.pspace.ifs.ksan.osd.utils.OSDConfig; -import com.pspace.ifs.ksan.osd.utils.OSDConstants; +import com.pspace.ifs.ksan.libs.Constants; import com.pspace.ifs.ksan.osd.utils.OSDUtils; +import com.pspace.ifs.ksan.osd.utils.OSDConstants; import com.pspace.ifs.ksan.libs.DiskManager; import com.pspace.ifs.ksan.libs.KsanUtils; +import com.pspace.ifs.ksan.libs.PrintStack; import com.pspace.ifs.ksan.libs.data.OsdData; +import com.pspace.ifs.ksan.libs.disk.Disk; +import com.pspace.ifs.ksan.libs.disk.DiskPool; +import com.pspace.ifs.ksan.libs.disk.Server; +import com.pspace.ifs.ksan.libs.OSDClient; +import com.pspace.ifs.ksan.libs.data.ECPart; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -34,6 +44,9 @@ public class DoECPriObject implements Runnable { private String localIP = KsanUtils.getLocalIP(); private long fileLength; private int ecWaitTime; + private int numberOfCodingChunks; + private int numberOfDataChunks; + HashMap localDiskInfoMap; @Override public void run() { @@ -41,41 +54,38 @@ public void run() { fileLength = OSDConfig.getInstance().getECMinSize() * OSDConstants.MEGABYTES; ecWaitTime = OSDConfig.getInstance().getECWaitTime(); - List diskList = new ArrayList(); - logger.debug(OSDConstants.LOG_DO_EC_PRI_OBJECT_LOCAL_IP, localIP); - HashMap diskInfoMap = DiskManager.getInstance().getLocalDiskInfo(); - diskInfoMap.forEach((diskId, diskPath) -> { - diskList.add(diskPath); + localDiskInfoMap = DiskManager.getInstance().getLocalDiskInfo(); + logger.debug("local disk info size : {}", localDiskInfoMap.size()); + localDiskInfoMap.forEach((diskId, diskPath) -> { + logger.debug("diskId: {}, diskPath: {}", diskId, diskPath); + numberOfCodingChunks = DiskManager.getInstance().getECM(diskId); + numberOfDataChunks = DiskManager.getInstance().getECK(diskId); + logger.debug("number of coding chunks: {}, number of data chunks: {}", numberOfCodingChunks, numberOfDataChunks); + if (numberOfCodingChunks > 0 && numberOfDataChunks > 0) { + // check EC + check(diskPath + Constants.SLASH + Constants.OBJ_DIR, diskPath + Constants.SLASH + Constants.EC_DIR); + } }); - - logger.debug(OSDConstants.LOG_DO_EC_PRI_OBJECT_DISKLIST_SIZE, diskList.size()); - for (String diskPath : diskList) { - String objPath = diskPath + OSDConstants.SLASH + OSDConstants.OBJ_DIR; - String ecPath = diskPath + OSDConstants.SLASH + OSDConstants.EC_DIR; - logger.debug(OSDConstants.LOG_DO_EC_PRI_OBJECT_PATH, objPath, ecPath); - - check(objPath, ecPath); - } } private void check(String dirPath, String ecPath) { File dir = new File(dirPath); File[] files = dir.listFiles(); long now = Calendar.getInstance().getTimeInMillis(); - for (int i = 0; i < files.length; i++) { if (files[i].isDirectory()) { - check(files[i].getPath(), ecPath); + check(files[i].getAbsolutePath(), ecPath); + continue; } - + logger.debug("file : {}", files[i].getName()); if (files[i].isFile()) { if (files[i].getName().startsWith(OSDConstants.POINT)) { continue; } - - if (OSDConstants.FILE_ATTRUBUTE_REPLICATION_PRIMARY.equals(OSDUtils.getInstance().getAttributeFileReplication(files[i]))) { - long diff = (now - files[i].lastModified()); // / OSDConstants.ONE_MINUTE_MILLISECONDS; - + String replica = KsanUtils.getAttributeFileReplication(files[i]); + logger.debug("replica : -{}-", replica); + if (Constants.FILE_ATTRUBUTE_REPLICATION_PRIMARY.equals(replica)) { + long diff = (now - files[i].lastModified()); if (diff >= ecWaitTime) { if (files[i].length() >= fileLength) { ecEncode(files[i], ecPath); @@ -89,30 +99,35 @@ private void check(String dirPath, String ecPath) { private void ecEncode(File file, String ecPath) { try { logger.info(OSDConstants.LOG_DO_EC_PRI_OBJECT_ENCODE_EC, file.getName()); - String path = OSDUtils.getInstance().makeECDirectory(file.getName(), ecPath); + String path = KsanUtils.makeECDirectory(file.getName(), ecPath); File ecFile = new File(path); com.google.common.io.Files.createParentDirs(ecFile); ecFile.mkdir(); - String command = OSDConstants.DO_EC_PRI_OBJECT_ZFEC + String command = Constants.ZFEC + path - + OSDConstants.DO_EC_PRI_OBJECT_ZFEC_PREFIX_OPTION + + Constants.ZFEC_PREFIX_OPTION + OSDConstants.POINT + file.getName() - + OSDConstants.DO_EC_PRI_OBJECT_ZFEC_TOTAL_NUMBER_OPTION + file.getAbsolutePath(); + + Constants.ZFEC_TOTAL_SHARES_OPTION + Integer.toString(numberOfCodingChunks + numberOfDataChunks) + + Constants.ZFEC_REQUIRED_SHARES_OPTION + Integer.toString(numberOfDataChunks) + Constants.SPACE + + file.getAbsolutePath(); logger.debug(OSDConstants.LOG_DO_EC_PRI_OBJECT_ZFEC_COMMAND, command); Process p = Runtime.getRuntime().exec(command); int exitCode = p.waitFor(); p.destroy(); logger.info(OSDConstants.LOG_DO_EC_PRI_OBJECT_ZFEC_EXIT_CODE, exitCode); - createECTemp(file.getName(), ecPath); - String replicaDiskID = OSDUtils.getInstance().getAttributeFileReplicaDiskID(file); - if (!OSDConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL.equals(replicaDiskID)) { + // spread ec file + spreadEC(path, file.getName()); + + String replicaDiskID = KsanUtils.getAttributeFileReplicaDiskID(file); + logger.debug("replica diskID : {}", replicaDiskID); + if (!Constants.FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL.equals(replicaDiskID)) { // delete replica disk logger.info(OSDConstants.LOG_DO_EC_PRI_OBJECT_REPLICA_DISK_ID, replicaDiskID); String ip = DiskManager.getInstance().getOSDIP(replicaDiskID); String diskPath = DiskManager.getInstance().getPath(replicaDiskID); - String replicaPath = OSDUtils.getInstance().makePath(diskPath, file.getName()); + String replicaPath = KsanUtils.makePath(diskPath, file.getName()); if (ip != null && diskPath != null) { if (localIP.equals(ip)) { // local replica @@ -132,32 +147,83 @@ private void ecEncode(File file, String ecPath) { } } - private void createECTemp(String fileName, String ecPath) { - // create ec temp file - File ecFile = new File(OSDUtils.getInstance().makeECTempPath(fileName, ecPath)); - try { - ecFile.createNewFile(); - } catch (IOException e) { - logger.error(e.getMessage()); - } - } - private void deleteReplica(String ipAddress, String replicaPath) { Socket socket = null; try { - socket = new Socket(ipAddress, OSDConfig.getInstance().getPort()); - socket.setTcpNoDelay(true); - String header = OsdData.DELETE_REPLICA + OsdData.DELIMITER + replicaPath; - logger.debug(OSDConstants.LOG_DO_EC_PRI_OBJECT_HEADER, header); - byte[] buffer = header.getBytes(OSDConstants.CHARSET_UTF_8); - int size = buffer.length; - - DataOutputStream so = new DataOutputStream(socket.getOutputStream()); - so.writeInt(size); - so.write(buffer, 0, size); - so.flush(); + OSDClient client = new OSDClient(ipAddress, OSDConfig.getInstance().getPort()); + client.deleteReplica(replicaPath); } catch (IOException e) { logger.error(e.getMessage()); } } + + private void spreadEC(String path, String fileName) { + File dir = new File(path); + File[] files = dir.listFiles(); + String[] ends = new String[files.length]; + File dest = new File(path + Constants.SLASH + Constants.POINT + fileName); + + logger.debug("ec parts : {}", files.length); + for (int i = 0; i < files.length; i++) { + ends[i] = Integer.toString(i) + Constants.UNDERSCORE + Integer.toString(numberOfCodingChunks + numberOfDataChunks) + Constants.ZFEC_SUFFIX; + } + + try { + List sendList = new ArrayList(); + for (DiskPool pool : DiskManager.getInstance().getDiskPoolList()) { + for (Server server : pool.getServerList()) { + for (Disk disk : server.getDiskList()) { + ECPart sendECPart = new ECPart(server.getIp(), disk.getPath(), false); + sendList.add(sendECPart); + } + } + } + + byte[] buffer = new byte[Constants.MAXBUFSIZE]; + for (int i = 0, j = 1; i < ends.length; i++) { + logger.debug("file : {}", files[i].getName()); + for (ECPart sendECPart : sendList) { + String ecPartPath = KsanUtils.makeECDirectory(files[i].getName().substring(1), sendECPart.getDiskPath() + Constants.SLASH + Constants.EC_DIR) + Constants.SLASH + Constants.POINT + fileName; + if (!sendECPart.isProcessed()) { + if (sendECPart.getServerIP().equals(localIP)) { + // if local disk, move file + try { + String copyPath = KsanUtils.makeECDirectory(files[i].getName().substring(1), sendECPart.getDiskPath() + Constants.SLASH + Constants.EC_DIR); + File file = new File(ecPartPath); + logger.debug("move file : {}, to : {}", files[i].getName(), file.getAbsolutePath()); + FileUtils.moveFile(files[i], file); + } catch (IOException e) { + PrintStack.logging(logger, e); + } + // FileUtils.copyFile(files[i], file); + sendECPart.setProcessed(true); + } else { + long fileLength = files[i].length(); + OSDClient client = new OSDClient(sendECPart.getServerIP(), OSDConfig.getInstance().getPort()); + logger.debug("send file : {}, to : {}, {}, {}", files[i].getName(), sendECPart.getServerIP(), sendECPart.getDiskPath(), ecPartPath); + client.putECPartInit(ecPartPath, fileLength); + int readLength = 0; + long totalReads = 0L; + try (FileInputStream fis = new FileInputStream(files[i])) { + while ((readLength = fis.read(buffer, 0, Constants.MAXBUFSIZE)) != -1) { + totalReads += readLength; + client.putECPart(buffer, 0, readLength); + if (totalReads >= fileLength) { + break; + } + } + } + client.putECPartFlush(); + sendECPart.setProcessed(true); + files[i].delete(); + } + break; + } + } + } + } catch (Exception e) { + PrintStack.logging(logger, e); + } + } } + diff --git a/core/src/com/pspace/ifs/ksan/osd/DoEmptyTrash.java b/core/src/com/pspace/ifs/ksan/osd/DoEmptyTrash.java index aa473db7..df689343 100644 --- a/core/src/com/pspace/ifs/ksan/osd/DoEmptyTrash.java +++ b/core/src/com/pspace/ifs/ksan/osd/DoEmptyTrash.java @@ -24,14 +24,15 @@ import com.pspace.ifs.ksan.osd.utils.OSDConstants; import com.pspace.ifs.ksan.osd.utils.OSDUtils; import com.pspace.ifs.ksan.libs.DiskManager; -import com.pspace.ifs.ksan.libs.KsanUtils; +import com.pspace.ifs.ksan.libs.KsanUtils; +import com.pspace.ifs.ksan.libs.Constants; public class DoEmptyTrash implements Runnable { private final static Logger logger = LoggerFactory.getLogger(DoEmptyTrash.class); @Override public void run() { - logger.info(OSDConstants.LOG_DO_EMPTY_TRASH_START); + // logger.info(OSDConstants.LOG_DO_EMPTY_TRASH_START); if (OSDConfig.getInstance().isCacheDiskpath()) { recursiveEmptyCache(OSDConfig.getInstance().getCacheDiskpath()); @@ -39,7 +40,7 @@ public void run() { HashMap diskInfoMap = DiskManager.getInstance().getLocalDiskInfo(); diskInfoMap.forEach((diskId, diskPath) -> { - String trashDir = diskPath + OSDConstants.SLASH + OSDConstants.TRASH_DIR; + String trashDir = diskPath + Constants.SLASH + Constants.TRASH_DIR; empty(trashDir); }); } @@ -50,7 +51,7 @@ private void recursiveEmptyCache(String dirPath) { for (int i = 0; i < files.length; i++) { if (files[i].isDirectory()) { - if (files[i].getName().equals(OSDConstants.TRASH_DIR)) { + if (files[i].getName().equals(Constants.TRASH_DIR)) { empty(files[i].getAbsolutePath()); } } diff --git a/core/src/com/pspace/ifs/ksan/osd/DoMoveCacheToDisk.java b/core/src/com/pspace/ifs/ksan/osd/DoMoveCacheToDisk.java index e699a398..b90d0837 100644 --- a/core/src/com/pspace/ifs/ksan/osd/DoMoveCacheToDisk.java +++ b/core/src/com/pspace/ifs/ksan/osd/DoMoveCacheToDisk.java @@ -17,6 +17,7 @@ import com.pspace.ifs.ksan.osd.utils.OSDConfig; import com.pspace.ifs.ksan.osd.utils.OSDConstants; import com.pspace.ifs.ksan.osd.utils.OSDUtils; +import com.pspace.ifs.ksan.libs.Constants; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -26,7 +27,7 @@ public class DoMoveCacheToDisk implements Runnable { @Override public void run() { - logger.info(OSDConstants.LOG_DO_MOVE_CACHE_TO_DISK); + // logger.info(OSDConstants.LOG_DO_MOVE_CACHE_TO_DISK); recursiveMove(OSDConfig.getInstance().getCacheDiskpath()); } @@ -36,7 +37,7 @@ private void recursiveMove(String dirPath) { for (int i = 0; i < files.length; i++) { if (files[i].isDirectory()) { - if (files[i].getName().equals(OSDConstants.OBJ_DIR)) { + if (files[i].getName().equals(Constants.OBJ_DIR)) { check(files[i].getAbsolutePath()); } else { recursiveMove(files[i].getAbsolutePath()); diff --git a/core/src/com/pspace/ifs/ksan/osd/EventObject.java b/core/src/com/pspace/ifs/ksan/osd/EventObject.java index abed5653..17c4d44f 100644 --- a/core/src/com/pspace/ifs/ksan/osd/EventObject.java +++ b/core/src/com/pspace/ifs/ksan/osd/EventObject.java @@ -91,14 +91,14 @@ public MQResponse call(String routingKey, String body) { logger.info("targetOsdIp : {}", targetOsdIp); byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; - String fullPath = OSDUtils.getInstance().makeObjPath(sourceDiskPath, objId, versionId); + String fullPath = KsanUtils.makeObjPath(sourceDiskPath, objId, versionId); logger.info("source full path : {}", fullPath); File srcFile = new File(fullPath); try (FileInputStream fis = new FileInputStream(srcFile)) { if (KsanUtils.getLocalIP().equals(targetOsdIp)) { - File file = new File(OSDUtils.getInstance().makeObjPath(targetDiskPath, objId, versionId)); - File tmpFile = new File(OSDUtils.getInstance().makeTempPath(targetDiskPath, objId, versionId)); - File trashFile = new File(OSDUtils.getInstance().makeTrashPath(targetDiskPath, objId, versionId)); + File file = new File(KsanUtils.makeObjPath(targetDiskPath, objId, versionId)); + File tmpFile = new File(KsanUtils.makeTempPath(targetDiskPath, objId, versionId)); + File trashFile = new File(KsanUtils.makeTrashPath(targetDiskPath, objId, versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); @@ -193,7 +193,7 @@ public MQResponse call(String routingKey, String body) { logger.info("diskId : {}", diskId); logger.info("diskPath : {}", diskPath); - String fullPath = OSDUtils.getInstance().makeObjPath(diskPath, objId, versionId); + String fullPath = KsanUtils.makeObjPath(diskPath, objId, versionId); logger.info("full path : {}", fullPath); File file = new File(fullPath); if (file.exists()) { @@ -257,7 +257,7 @@ public MQResponse call(String routingKey, String body) { logger.info("diskId : {}", diskId); logger.info("diskPath : {}", diskPath); - String fullPath = OSDUtils.getInstance().makeObjPath(diskPath, objId, versionId); + String fullPath = KsanUtils.makeObjPath(diskPath, objId, versionId); logger.info("full path : {}", fullPath); File file = new File(fullPath); @@ -343,14 +343,14 @@ public MQResponse call(String routingKey, String body) { logger.info("targetOsdIp : {}", targetOsdIp); byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; - String fullPath = OSDUtils.getInstance().makeObjPath(sourceDiskPath, objId, versionId); + String fullPath = KsanUtils.makeObjPath(sourceDiskPath, objId, versionId); logger.info("full path : {}", fullPath); File srcFile = new File(fullPath); try (FileInputStream fis = new FileInputStream(srcFile)) { if (KsanUtils.getLocalIP().equals(targetOsdIp)) { - File file = new File(OSDUtils.getInstance().makeObjPath(targetDiskPath, objId, versionId)); - File tmpFile = new File(OSDUtils.getInstance().makeTempPath(targetDiskPath, objId, versionId)); - File trashFile = new File(OSDUtils.getInstance().makeTrashPath(targetDiskPath, objId, versionId)); + File file = new File(KsanUtils.makeObjPath(targetDiskPath, objId, versionId)); + File tmpFile = new File(KsanUtils.makeTempPath(targetDiskPath, objId, versionId)); + File trashFile = new File(KsanUtils.makeTrashPath(targetDiskPath, objId, versionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); diff --git a/core/src/com/pspace/ifs/ksan/osd/OSDPortal.java b/core/src/com/pspace/ifs/ksan/osd/OSDPortal.java index 0b2e6e0d..0fbf5f27 100644 --- a/core/src/com/pspace/ifs/ksan/osd/OSDPortal.java +++ b/core/src/com/pspace/ifs/ksan/osd/OSDPortal.java @@ -240,7 +240,24 @@ public void getDiskPoolsDetails() { for (int i = 0; i < jsonItems.size(); i++) { JSONObject item = (JSONObject)jsonItems.get(i); - DiskPool diskPool = new DiskPool((String)item.get(DiskPool.ID), (String)item.get(DiskPool.NAME), (String)item.get(DiskPool.DISK_POOL_TYPE), (String)item.get(DiskPool.REPLICATION_TYPE)); + JSONObject jsonEC = (JSONObject)item.get(DiskPool.EC); + + DiskPool diskPool = null; + if (jsonEC != null) { + logger.info("jsonEC : {}", jsonEC.toString()); + diskPool = new DiskPool((String)item.get(DiskPool.ID), + (String)item.get(DiskPool.NAME), + (String)item.get(DiskPool.DISK_POOL_TYPE), + (String)item.get(DiskPool.REPLICATION_TYPE), + (int)(long)jsonEC.get(DiskPool.EC_M), + (int)(long)jsonEC.get(DiskPool.EC_K)); + } else { + diskPool = new DiskPool((String)item.get(DiskPool.ID), + (String)item.get(DiskPool.NAME), + (String)item.get(DiskPool.DISK_POOL_TYPE), + (String)item.get(DiskPool.REPLICATION_TYPE)); + } + // DiskPool diskPool = new DiskPool((String)item.get(DiskPool.ID), (String)item.get(DiskPool.NAME), (String)item.get(DiskPool.DISK_POOL_TYPE), (String)item.get(DiskPool.REPLICATION_TYPE)); JSONArray jsonServers = (JSONArray)item.get(DiskPool.SERVERS); if (jsonServers != null && jsonServers.size() == 0) { logger.info("diskpools -- servers is empty"); diff --git a/core/src/com/pspace/ifs/ksan/osd/OSDServer.java b/core/src/com/pspace/ifs/ksan/osd/OSDServer.java index cd9e2d61..67409da7 100644 --- a/core/src/com/pspace/ifs/ksan/osd/OSDServer.java +++ b/core/src/com/pspace/ifs/ksan/osd/OSDServer.java @@ -50,10 +50,12 @@ import com.pspace.ifs.ksan.libs.HeartbeatManager; import com.pspace.ifs.ksan.libs.config.AgentConfig; import com.pspace.ifs.ksan.libs.OSDClient; +import com.pspace.ifs.ksan.libs.Constants; import java.util.concurrent.TimeUnit; import java.util.concurrent.ScheduledExecutorService; +import de.sfuhrm.openssl4j.OpenSSL4JProvider; import org.apache.commons.crypto.stream.CtrCryptoInputStream; import org.apache.commons.crypto.stream.CtrCryptoOutputStream; import org.slf4j.Logger; @@ -96,6 +98,10 @@ public void start() { PrintStack.logging(logger, e); } + // startEmptyTrash(); + // startMoveCacheToDisk(); + startECThread(); + int poolSize = OSDConfig.getInstance().getPoolSize(); localIP = KsanUtils.getLocalIP(); port = OSDConfig.getInstance().getPort(); @@ -129,7 +135,7 @@ private class Worker implements Runnable { public void run() { try { byte[] buffer = new byte[OSDConstants.HEADERSIZE]; - DataInputStream di = new DataInputStream(socket.getInputStream()); + // DataInputStream di = new DataInputStream(socket.getInputStream()); boolean flag = false; while (true) { @@ -203,6 +209,17 @@ public void run() { case OsdData.STOP: isRunning = false; break; + + case OsdData.GET_EC_PART: + getECPart(headers); + break; + + case OsdData.PUT_EC_PART: + putECPart(headers); + break; + + case OsdData.DELETE_EC_PART: + deleteECPart(headers); default: logger.error(OSDConstants.LOG_OSD_SERVER_UNKNOWN_INDICATOR, indicator); @@ -215,16 +232,15 @@ public void run() { } } catch (IOException | NoSuchAlgorithmException e) { logger.info("socket : {} - {}", socket.getRemoteSocketAddress().toString(), e.getMessage()); - - if (socket.isClosed()) { - logger.error("Socket is closed"); - } - if (!socket.isConnected()) { - logger.error("Socket is not connected"); - } - if (socket.isInputShutdown()) { - logger.error("Socket input is shutdown"); - } + // if (socket.isClosed()) { + // logger.error("Socket is closed"); + // } + // if (!socket.isConnected()) { + // logger.error("Socket is not connected"); + // } + // if (socket.isInputShutdown()) { + // logger.error("Socket input is shutdown"); + // } } finally { try { socket.close(); @@ -247,7 +263,7 @@ private void get(String[] headers) throws IOException { logger.debug(OSDConstants.LOG_OSD_SERVER_GET_INFO, path, objId, versionId, sourceRange); byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; - File file = new File(OSDUtils.getInstance().makeObjPath(path, objId, versionId)); + File file = new File(KsanUtils.makeObjPath(path, objId, versionId)); if (key.equalsIgnoreCase(OSDConstants.STR_NULL)) { key = null; @@ -355,6 +371,38 @@ private void get(String[] headers) throws IOException { logger.info("from : {}", socket.getRemoteSocketAddress().toString()); logger.info(OSDConstants.LOG_OSD_SERVER_GET_SUCCESS_INFO, path, objId, versionId, sourceRange); } + + private void getECPart(String[] headers) { + logger.debug(OSDConstants.LOG_OSD_SERVER_GET_EC_PART_START); + String path = headers[OsdData.PATH_INDEX]; + logger.debug("path : {}", path); + byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; + File file = new File(path); + + try (FileInputStream fis = new FileInputStream(file)) { + long remainLength = file.length(); + int readLength = 0; + int readBytes; + + while (remainLength > 0) { + readBytes = 0; + if (remainLength < OSDConstants.MAXBUFSIZE) { + readBytes = (int)remainLength; + } else { + readBytes = OSDConstants.MAXBUFSIZE; + } + readLength = fis.read(buffer, 0, readBytes); + socket.getOutputStream().write(buffer, 0, readLength); + remainLength -= readLength; + } + socket.getOutputStream().flush(); + socket.getOutputStream().close(); + } catch (IOException e) { + PrintStack.logging(logger, e); + } + + logger.debug(OSDConstants.LOG_OSD_SERVER_GET_EC_PART_END); + } private void put(String[] headers) throws IOException { logger.debug(OSDConstants.LOG_OSD_SERVER_PUT_START); @@ -387,15 +435,15 @@ private void put(String[] headers) throws IOException { } if (!Strings.isNullOrEmpty(key)) { if (OSDConfig.getInstance().isCacheDiskpath()) { - file = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeObjPath(path, objId, versionId))); - tmpFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTempPath(path, objId, versionId))); - trashFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTrashPath(path, objId, versionId))); - File linkFile = new File(OSDUtils.getInstance().makeObjPath(path, objId, versionId)); + file = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeObjPath(path, objId, versionId)); + tmpFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTempPath(path, objId, versionId)); + trashFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTrashPath(path, objId, versionId)); + File linkFile = new File(KsanUtils.makeObjPath(path, objId, versionId)); com.google.common.io.Files.createParentDirs(linkFile); } else { - file = new File(OSDUtils.getInstance().makeObjPath(path, objId, versionId)); - tmpFile = new File(OSDUtils.getInstance().makeTempPath(path, objId, versionId)); - trashFile = new File(OSDUtils.getInstance().makeTrashPath(path, objId, versionId)); + file = new File(KsanUtils.makeObjPath(path, objId, versionId)); + tmpFile = new File(KsanUtils.makeTempPath(path, objId, versionId)); + trashFile = new File(KsanUtils.makeTrashPath(path, objId, versionId)); } com.google.common.io.Files.createParentDirs(file); @@ -421,15 +469,15 @@ private void put(String[] headers) throws IOException { } } else { if (OSDConfig.getInstance().isCacheDiskpath()) { - file = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeObjPath(path, objId, versionId))); - tmpFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTempPath(path, objId, versionId))); - trashFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTrashPath(path, objId, versionId))); - File linkFile = new File(OSDUtils.getInstance().makeObjPath(path, objId, versionId)); + file = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeObjPath(path, objId, versionId)); + tmpFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTempPath(path, objId, versionId)); + trashFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTrashPath(path, objId, versionId)); + File linkFile = new File(KsanUtils.makeObjPath(path, objId, versionId)); com.google.common.io.Files.createParentDirs(linkFile); } else { - file = new File(OSDUtils.getInstance().makeObjPath(path, objId, versionId)); - tmpFile = new File(OSDUtils.getInstance().makeTempPath(path, objId, versionId)); - trashFile = new File(OSDUtils.getInstance().makeTrashPath(path, objId, versionId)); + file = new File(KsanUtils.makeObjPath(path, objId, versionId)); + tmpFile = new File(KsanUtils.makeTempPath(path, objId, versionId)); + trashFile = new File(KsanUtils.makeTrashPath(path, objId, versionId)); } com.google.common.io.Files.createParentDirs(file); @@ -460,16 +508,40 @@ private void put(String[] headers) throws IOException { retryRenameTo(temp, trashFile); } - // OSDUtils.getInstance().setAttributeFileReplication(tmpFile, replication, replicaDiskID); + KsanUtils.setAttributeFileReplication(tmpFile, replication, replicaDiskID); retryRenameTo(tmpFile, file); if (OSDConfig.getInstance().isCacheDiskpath()) { - String fullPath = OSDUtils.getInstance().makeObjPath(path, objId, versionId); + String fullPath = KsanUtils.makeObjPath(path, objId, versionId); Files.createSymbolicLink(Paths.get(fullPath), Paths.get(file.getAbsolutePath())); } logger.debug(OSDConstants.LOG_OSD_SERVER_PUT_END); logger.info("from : {}", socket.getRemoteSocketAddress().toString()); logger.info(OSDConstants.LOG_OSD_SERVER_PUT_SUCCESS_INFO, path, objId, versionId, length); } + + private void putECPart(String[] headers) throws IOException { + logger.debug(OSDConstants.LOG_OSD_SERVER_PUT_EC_PART_START); + String path = headers[OsdData.PATH_INDEX]; + long length = Longs.tryParse(headers[OsdData.PUT_EC_LENGTH_INDEX]); + + byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; + File file = new File(path); + com.google.common.io.Files.createParentDirs(file); + try (FileOutputStream fos = new FileOutputStream(file, false)) { + int readLength = 0; + long remainLength = length; + int readMax = (int) (length < OSDConstants.MAXBUFSIZE ? length : OSDConstants.MAXBUFSIZE); + while ((readLength = socket.getInputStream().read(buffer, 0, readMax)) != -1) { + remainLength -= readLength; + fos.write(buffer, 0, readLength); + if (remainLength <= 0) { + break; + } + readMax = (int) (remainLength < OSDConstants.MAXBUFSIZE ? remainLength : OSDConstants.MAXBUFSIZE); + } + } + logger.debug(OSDConstants.LOG_OSD_SERVER_PUT_EC_PART_END); + } private void delete(String[] headers) throws IOException { logger.debug(OSDConstants.LOG_OSD_SERVER_DELETE_START); @@ -481,22 +553,22 @@ private void delete(String[] headers) throws IOException { File file = null; File trashFile = null; if (OSDConfig.getInstance().isCacheDiskpath()) { - file = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeObjPath(path, objId, versionId))); + file = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeObjPath(path, objId, versionId)); if (file.exists()) { isCache = true; - trashFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTrashPath(path, objId, versionId))); + trashFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTrashPath(path, objId, versionId)); } else { - file = new File(OSDUtils.getInstance().makeObjPath(path, objId, versionId)); - trashFile = new File(OSDUtils.getInstance().makeTrashPath(path, objId, versionId)); + file = new File(KsanUtils.makeObjPath(path, objId, versionId)); + trashFile = new File(KsanUtils.makeTrashPath(path, objId, versionId)); } } else { - file = new File(OSDUtils.getInstance().makeObjPath(path, objId, versionId)); - trashFile = new File(OSDUtils.getInstance().makeTrashPath(path, objId, versionId)); + file = new File(KsanUtils.makeObjPath(path, objId, versionId)); + trashFile = new File(KsanUtils.makeTrashPath(path, objId, versionId)); } retryRenameTo(file, trashFile); if (isCache) { - File link = new File(OSDUtils.getInstance().makeObjPath(path, objId, versionId)); + File link = new File(KsanUtils.makeObjPath(path, objId, versionId)); link.delete(); } @@ -511,15 +583,31 @@ private void deletePart(String[] headers) throws IOException { String partNo = headers[OsdData.PARTNO_INDEX]; logger.debug(OSDConstants.LOG_OSD_SERVER_DELETE_PART_INFO, path, objId, partNo); - File file = new File(OSDUtils.getInstance().makeTempPath(path, objId, partNo)); + File file = new File(KsanUtils.makeTempPartPath(path, objId, partNo)); if (file.exists()) { file.delete(); + } else { + logger.error("does not exist: {}", file.getAbsolutePath()); } logger.debug(OSDConstants.LOG_OSD_SERVER_DELETE_PART_END); logger.info(OSDConstants.LOG_OSD_SERVER_DELETE_PART_SUCCESS_INFO, path, objId, partNo); } + private void deleteECPart(String[] headers) { + logger.debug(OSDConstants.LOG_OSD_SERVER_DELETE_EC_PART_START); + String path = headers[OsdData.PATH_INDEX]; + + File file = new File(path); + if (file.exists()) { + file.delete(); + logger.debug("ec part delete : {}", path); + } else { + logger.debug("file does not exist. : {}", path); + } + logger.debug(OSDConstants.LOG_OSD_SERVER_DELETE_EC_PART_END); + } + private void deleteReplica(String[] headers) throws IOException { logger.debug(OSDConstants.LOG_OSD_SERVER_DELETE_REPLICA_START); String path = headers[OsdData.PATH_INDEX]; @@ -528,6 +616,8 @@ private void deleteReplica(String[] headers) throws IOException { File file = new File(path); if (file.exists()) { file.delete(); + } else { + logger.info("deleteReplica, does not exist: {}", path); } logger.debug(OSDConstants.LOG_OSD_SERVER_DELETE_REPLICA_END); @@ -553,12 +643,12 @@ private void copy(String[] headers) throws IOException, NoSuchAlgorithmException } byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; - File srcFile = new File(OSDUtils.getInstance().makeObjPath(srcPath, srcObjId, srcVersionId)); + File srcFile = new File(KsanUtils.makeObjPath(srcPath, srcObjId, srcVersionId)); try (FileInputStream fis = new FileInputStream(srcFile)) { if (localIP.equals(destIP)) { - File file = new File(OSDUtils.getInstance().makeObjPath(destPath, destObjId, destVersionId)); - File tmpFile = new File(OSDUtils.getInstance().makeTempPath(destPath, destObjId, destVersionId)); - File trashFile = new File(OSDUtils.getInstance().makeTrashPath(destPath, destObjId, destVersionId)); + File file = new File(KsanUtils.makeObjPath(destPath, destObjId, destVersionId)); + File tmpFile = new File(KsanUtils.makeTempPath(destPath, destObjId, destVersionId)); + File trashFile = new File(KsanUtils.makeTrashPath(destPath, destObjId, destVersionId)); com.google.common.io.Files.createParentDirs(file); com.google.common.io.Files.createParentDirs(tmpFile); @@ -573,7 +663,7 @@ private void copy(String[] headers) throws IOException, NoSuchAlgorithmException File temp = new File(file.getAbsolutePath()); retryRenameTo(temp, trashFile); } - // OSDUtils.getInstance().setAttributeFileReplication(file, replication, replicaDiskID); + KsanUtils.setAttributeFileReplication(file, replication, replicaDiskID); retryRenameTo(tmpFile, file); } else { try (Socket destSocket = new Socket(destIP, port)) { @@ -586,7 +676,7 @@ private void copy(String[] headers) throws IOException, NoSuchAlgorithmException + OsdData.DELIMITER + replicaDiskID; logger.debug(OSDConstants.LOG_OSD_SERVER_COPY_RELAY_OSD, destIP, header); sendHeader(destSocket, header); - MessageDigest md5er = MessageDigest.getInstance(OSDConstants.MD5); + MessageDigest md5er = MessageDigest.getInstance(OSDConstants.MD5, new OpenSSL4JProvider()); int readLength = 0; while ((readLength = fis.read(buffer, 0, OSDConstants.MAXBUFSIZE)) != -1) { @@ -621,9 +711,9 @@ private void getPart(String[] headers) throws IOException { byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; File file = null; if (OSDConfig.getInstance().isCacheDiskpath()) { - file = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTempPartPath(path, objId, partNo))); + file = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTempPartPath(path, objId, partNo)); } else { - file = new File(OSDUtils.getInstance().makeTempPartPath(path, objId, partNo)); + file = new File(KsanUtils.makeTempPartPath(path, objId, partNo)); } try (FileInputStream fis = new FileInputStream(file)) { long remainLength = 0L; @@ -631,6 +721,7 @@ private void getPart(String[] headers) throws IOException { int readBytes; remainLength = file.length(); + logger.debug("file length : {}", remainLength); while (remainLength > 0) { readBytes = 0; @@ -649,9 +740,9 @@ private void getPart(String[] headers) throws IOException { logger.info(OSDConstants.LOG_OSD_SERVER_GET_PART_SUCCESS_INFO, path, objId, partNo); socket.getOutputStream().flush(); - if (!file.delete()) { - logger.error(OSDConstants.LOG_OSD_SERVER_FAILED_FILE_DELETE, file.getName()); - } + // if (!file.delete()) { + // logger.error(OSDConstants.LOG_OSD_SERVER_FAILED_FILE_DELETE, file.getName()); + // } } private void part(String[] headers) throws IOException { @@ -668,9 +759,9 @@ private void part(String[] headers) throws IOException { byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; File tmpFile = null; if (OSDConfig.getInstance().isCacheDiskpath()) { - tmpFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTempPartPath(path, objId, partNo))); + tmpFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTempPartPath(path, objId, partNo)); } else { - tmpFile = new File(OSDUtils.getInstance().makeTempPartPath(path, objId, partNo)); + tmpFile = new File(KsanUtils.makeTempPartPath(path, objId, partNo)); } com.google.common.io.Files.createParentDirs(tmpFile); @@ -716,72 +807,48 @@ private void part(String[] headers) throws IOException { private void partCopy(String[] headers) throws IOException, NoSuchAlgorithmException { logger.debug(OSDConstants.LOG_OSD_SERVER_PART_COPY_START); - String srcPath = headers[OsdData.PATH_INDEX]; + String srcDiskId = headers[OsdData.PATH_INDEX]; String srcObjId = headers[OsdData.OBJID_INDEX]; String srcVersionId = headers[OsdData.VERSIONID_INDEX]; String destPath = headers[OsdData.DEST_PATH_INDEX]; String destObjId = headers[OsdData.DEST_OBJID_INDEX]; String destPartNo = headers[OsdData.DEST_PARTNO_INDEX]; String copySourceRange = headers[OsdData.SRC_RANAGE_INDEX]; + String srcLength = headers[OsdData.SRC_LENGTH_INDEX]; - logger.debug(OSDConstants.LOG_OSD_SERVER_PART_COPY_INFO, srcPath, srcObjId, srcVersionId, destPath, destObjId, destPartNo, copySourceRange); + logger.debug(OSDConstants.LOG_OSD_SERVER_PART_COPY_INFO, srcDiskId, srcObjId, srcVersionId, destPath, destObjId, destPartNo, copySourceRange); - String destIP = null; //findOSD(destPath); - if (destIP == null) { - logger.error(OSDConstants.LOG_OSD_SERVER_CAN_NOT_FIND_OSD_IP, destPath); - return; - } + // String destIP = null; //findOSD(destPath); + // if (destIP == null) { + // logger.error(OSDConstants.LOG_OSD_SERVER_CAN_NOT_FIND_OSD_IP, destPath); + // return; + // } - MessageDigest md5er = MessageDigest.getInstance(OSDConstants.MD5); + MessageDigest md5er = MessageDigest.getInstance(OSDConstants.MD5, new OpenSSL4JProvider()); byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; - File srcFile = new File(OSDUtils.getInstance().makeObjPath(srcPath, srcObjId, srcVersionId)); + File srcFile = null; //new File(KsanUtils.makeObjPath(srcPath, srcObjId, srcVersionId)); long remainLength = 0L; int readLength = 0; - int readBytes; + int readBytes = 0; + long readTotal = 0L; String eTag = ""; OsdData data = null; - try (FileInputStream fis = new FileInputStream(srcFile)) { - File tmpFile = null; - if (OSDConfig.getInstance().isCacheDiskpath()) { - tmpFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTempPath(destPath, destObjId, destPartNo))); - } else { - tmpFile = new File(OSDUtils.getInstance().makeTempPath(destPath, destObjId, destPartNo)); - } - com.google.common.io.Files.createParentDirs(tmpFile); - try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { - data = new OsdData(); - if (Strings.isNullOrEmpty(copySourceRange)) { - remainLength = srcFile.length(); - data.setFileSize(remainLength); - while (remainLength > 0) { - readBytes = 0; - if (remainLength < OSDConstants.MAXBUFSIZE) { - readBytes = (int)remainLength; - } else { - readBytes = OSDConstants.MAXBUFSIZE; - } - readLength = fis.read(buffer, 0, readBytes); - fos.write(buffer, 0, readLength); - md5er.update(buffer, 0, readLength); - remainLength -= readLength; - } - fos.flush(); - } else { - String[] ranges = copySourceRange.split(OSDConstants.SLASH); - long totalLength = 0L; - for (String range : ranges) { - String[] rangeParts = range.split(OSDConstants.COMMA); - long offset = Longs.tryParse(rangeParts[OSDConstants.RANGE_OFFSET_INDEX]); - long length = Longs.tryParse(rangeParts[OSDConstants.RANGE_LENGTH_INDEX]); - logger.debug(OSDConstants.LOG_OSD_SERVER_RANGE_INFO, offset, length); - - if (offset > 0) { - fis.skip(offset); - } - remainLength = length; - totalLength += length; + File tmpFile = null; + if (OSDConfig.getInstance().isCacheDiskpath()) { + tmpFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTempPartPath(destPath, destObjId, destPartNo)); + } else { + tmpFile = new File(KsanUtils.makeTempPartPath(destPath, destObjId, destPartNo)); + } + + try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { + String srcPath = DiskManager.getInstance().getLocalPath(srcDiskId); + if (srcPath != null) { + srcFile = new File(KsanUtils.makeObjPath(srcPath, srcObjId, srcVersionId)); + try (FileInputStream fis = new FileInputStream(srcFile)) { + if (Strings.isNullOrEmpty(copySourceRange)) { + remainLength = srcFile.length(); while (remainLength > 0) { readBytes = 0; if (remainLength < OSDConstants.MAXBUFSIZE) { @@ -789,24 +856,124 @@ private void partCopy(String[] headers) throws IOException, NoSuchAlgorithmExcep } else { readBytes = OSDConstants.MAXBUFSIZE; } + readLength = fis.read(buffer, 0, readBytes); + readTotal += readLength; fos.write(buffer, 0, readLength); md5er.update(buffer, 0, readLength); remainLength -= readLength; } - fos.flush(); - - data.setFileSize(totalLength); + } else { + String[] ranges = copySourceRange.split(OSDConstants.SLASH); + for (String range : ranges) { + String[] rangeParts = range.split(OSDConstants.COMMA); + long offset = Longs.tryParse(rangeParts[OSDConstants.RANGE_OFFSET_INDEX]); + long length = Longs.tryParse(rangeParts[OSDConstants.RANGE_LENGTH_INDEX]); + logger.debug(OSDConstants.LOG_OSD_SERVER_RANGE_INFO, offset, length); + + if (offset > 0) { + fis.skip(offset); + } + remainLength = length; + while (remainLength > 0) { + readBytes = 0; + if (remainLength < OSDConstants.MAXBUFSIZE) { + readBytes = (int)remainLength; + } else { + readBytes = OSDConstants.MAXBUFSIZE; + } + + readLength = fis.read(buffer, 0, readBytes); + readTotal += readLength; + fos.write(buffer, 0, readLength); + md5er.update(buffer, 0, readLength); + remainLength -= readLength; + } + } } } + } else { + OSDClient client = new OSDClient(DiskManager.getInstance().getOSDIP(srcDiskId), OSDConfig.getInstance().getPort()); + client.getInitWithMD5(DiskManager.getInstance().getPath(srcDiskId), + srcObjId, + srcVersionId, + Long.parseLong(srcLength), + copySourceRange, + fos, + md5er, + ""); + readTotal = client.getWithMD5(); } - byte[] digest = md5er.digest(); - eTag = base16().lowerCase().encode(digest); } - sendData(data.getETag(), data.getFileSize()); + byte[] digest = md5er.digest(); + eTag = base16().lowerCase().encode(digest); + + // try (FileInputStream fis = new FileInputStream(srcFile)) { + // File tmpFile = null; + // if (OSDConfig.getInstance().isCacheDiskpath()) { + // tmpFile = new File(KsanUtils.makeCachePath(KsanUtils.makeTempPath(destPath, destObjId, destPartNo))); + // } else { + // tmpFile = new File(KsanUtils.makeTempPath(destPath, destObjId, destPartNo)); + // } + // com.google.common.io.Files.createParentDirs(tmpFile); + // try (FileOutputStream fos = new FileOutputStream(tmpFile, false)) { + // data = new OsdData(); + // if (Strings.isNullOrEmpty(copySourceRange)) { + // remainLength = srcFile.length(); + // data.setFileSize(remainLength); + // while (remainLength > 0) { + // readBytes = 0; + // if (remainLength < OSDConstants.MAXBUFSIZE) { + // readBytes = (int)remainLength; + // } else { + // readBytes = OSDConstants.MAXBUFSIZE; + // } + // readLength = fis.read(buffer, 0, readBytes); + // fos.write(buffer, 0, readLength); + // md5er.update(buffer, 0, readLength); + // remainLength -= readLength; + // } + // fos.flush(); + // } else { + // String[] ranges = copySourceRange.split(OSDConstants.SLASH); + // long totalLength = 0L; + // for (String range : ranges) { + // String[] rangeParts = range.split(OSDConstants.COMMA); + // long offset = Longs.tryParse(rangeParts[OSDConstants.RANGE_OFFSET_INDEX]); + // long length = Longs.tryParse(rangeParts[OSDConstants.RANGE_LENGTH_INDEX]); + // logger.debug(OSDConstants.LOG_OSD_SERVER_RANGE_INFO, offset, length); + + // if (offset > 0) { + // fis.skip(offset); + // } + // remainLength = length; + // totalLength += length; + // while (remainLength > 0) { + // readBytes = 0; + // if (remainLength < OSDConstants.MAXBUFSIZE) { + // readBytes = (int)remainLength; + // } else { + // readBytes = OSDConstants.MAXBUFSIZE; + // } + // readLength = fis.read(buffer, 0, readBytes); + // fos.write(buffer, 0, readLength); + // md5er.update(buffer, 0, readLength); + // remainLength -= readLength; + // } + // fos.flush(); + + // data.setFileSize(totalLength); + // } + // } + // } + // byte[] digest = md5er.digest(); + // eTag = base16().lowerCase().encode(digest); + // } + logger.debug("etag : {}, readTotal : {}", eTag, readTotal); + sendData(eTag, readTotal); logger.debug(OSDConstants.LOG_OSD_SERVER_PART_COPY_END); - logger.info(OSDConstants.LOG_OSD_SERVER_PART_COPY_SUCCESS_INFO, srcPath, srcObjId, srcVersionId, destPath, destObjId, destPartNo, copySourceRange); + // logger.info(OSDConstants.LOG_OSD_SERVER_PART_COPY_SUCCESS_INFO, srcPath, srcObjId, srcVersionId, destPath, destObjId, destPartNo, copySourceRange); } private void completeMultipart(String[] headers) throws IOException, NoSuchAlgorithmException { @@ -815,18 +982,20 @@ private void completeMultipart(String[] headers) throws IOException, NoSuchAlgor String objId = headers[OsdData.OBJID_INDEX]; String versionId = headers[OsdData.VERSIONID_INDEX]; String key = headers[OsdData.COMPLETE_MULTIPART_KEY_INDEX]; + String replication = headers[OsdData.COMPLETE_MULTIPART_REPLICATION_INDEX]; + String replicaDiskId = headers[OsdData.COMPLETE_MULTIPART_REPLICA_DISKID_INDEX]; String partInfos = headers[OsdData.COMPLETE_MULTIPART_PARTNOS_INDEX]; logger.debug(OSDConstants.LOG_OSD_SERVER_COMPLETE_MULTIPART_INFO, path, objId, versionId, key, partInfos); byte[] buffer = new byte[OSDConstants.MAXBUFSIZE]; - MessageDigest md5er = MessageDigest.getInstance(OSDConstants.MD5); + MessageDigest md5er = MessageDigest.getInstance(OSDConstants.MD5, new OpenSSL4JProvider()); long totalLength = 0L; long existFileSize = 0L; long putSize = 0L; long calSize = 0L; CtrCryptoOutputStream encryptOS = null; - CtrCryptoInputStream encryptIS = null; - String eTag; + // CtrCryptoInputStream encryptIS = null; + String eTag = ""; String[] arrayPartInfo = partInfos.split(OSDConstants.COMMA); @@ -849,13 +1018,13 @@ private void completeMultipart(String[] headers) throws IOException, NoSuchAlgor File trashFile = null; if (OSDConfig.getInstance().isCacheDiskpath()) { - file = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeObjPath(path, objId, versionId))); - tmpFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTempPath(path, objId, versionId))); - trashFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTrashPath(path, objId, versionId))); + file = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeObjPath(path, objId, versionId)); + tmpFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTempPath(path, objId, versionId)); + trashFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTrashPath(path, objId, versionId)); } else { - file = new File(OSDUtils.getInstance().makeObjPath(path, objId, versionId)); - tmpFile = new File(OSDUtils.getInstance().makeTempPath(path, objId, versionId)); - trashFile = new File(OSDUtils.getInstance().makeTrashPath(path, objId, versionId)); + file = new File(KsanUtils.makeObjPath(path, objId, versionId)); + tmpFile = new File(KsanUtils.makeTempPath(path, objId, versionId)); + trashFile = new File(KsanUtils.makeTrashPath(path, objId, versionId)); } com.google.common.io.Files.createParentDirs(tmpFile); com.google.common.io.Files.createParentDirs(file); @@ -876,15 +1045,15 @@ private void completeMultipart(String[] headers) throws IOException, NoSuchAlgor logger.debug("key : {}, part : {}, diskID : {}, part path : {}", key, entry.getKey(), entry.getValue().getDiskID(), partPath); File partFile = null; if (OSDConfig.getInstance().isCacheDiskpath()) { - partFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTempPartPath(partPath, objId, String.valueOf(entry.getValue().getPartNumber())))); + partFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTempPartPath(partPath, objId, String.valueOf(entry.getValue().getPartNumber()))); } else { - partFile = new File(OSDUtils.getInstance().makeTempPartPath(partPath, objId, String.valueOf(entry.getValue().getPartNumber()))); + partFile = new File(KsanUtils.makeTempPartPath(partPath, objId, String.valueOf(entry.getValue().getPartNumber()))); } try (FileInputStream fis = new FileInputStream(partFile)) { - encryptIS = OSDUtils.initCtrDecrypt(fis, key); + // encryptIS = OSDUtils.initCtrDecrypt(fis, key); int readLength = 0; - while ((readLength = encryptIS.read(buffer, 0, OSDConstants.MAXBUFSIZE)) != -1) { + while ((readLength = fis.read(buffer, 0, OSDConstants.MAXBUFSIZE)) != -1) { totalLength += readLength; encryptOS.write(buffer, 0, readLength); md5er.update(buffer, 0, readLength); @@ -909,9 +1078,9 @@ private void completeMultipart(String[] headers) throws IOException, NoSuchAlgor logger.debug("part : {}, diskID : {}, part path : {}", entry.getKey(), entry.getValue().getDiskID(), partPath); File partFile = null; if (OSDConfig.getInstance().isCacheDiskpath()) { - partFile = new File(OSDUtils.getInstance().makeCachePath(OSDUtils.getInstance().makeTempPartPath(partPath, objId, String.valueOf(entry.getValue().getPartNumber())))); + partFile = new File(OSDConfig.getInstance().getCacheDiskpath() + KsanUtils.makeTempPartPath(partPath, objId, String.valueOf(entry.getValue().getPartNumber()))); } else { - partFile = new File(OSDUtils.getInstance().makeTempPartPath(partPath, objId, String.valueOf(entry.getValue().getPartNumber()))); + partFile = new File(KsanUtils.makeTempPartPath(partPath, objId, String.valueOf(entry.getValue().getPartNumber()))); } try (FileInputStream fis = new FileInputStream(partFile)) { int readLength = 0; @@ -935,17 +1104,22 @@ private void completeMultipart(String[] headers) throws IOException, NoSuchAlgor byte[] digest = md5er.digest(); eTag = base16().lowerCase().encode(digest); + } catch (Exception e) { + PrintStack.logging(logger, e); } + logger.info("total length : {}", totalLength); + if (file.exists()) { File temp = new File(file.getAbsolutePath()); logger.info("file is already exists : {}", file.getAbsolutePath()); retryRenameTo(temp, trashFile); } + KsanUtils.setAttributeFileReplication(tmpFile, replication, replicaDiskId); retryRenameTo(tmpFile, file); if (OSDConfig.getInstance().isCacheDiskpath()) { - String fullPath = OSDUtils.getInstance().makeObjPath(path, objId, versionId); + String fullPath = KsanUtils.makeObjPath(path, objId, versionId); Files.createSymbolicLink(Paths.get(fullPath), Paths.get(file.getAbsolutePath())); } @@ -962,7 +1136,7 @@ private void abortMultipart(String[] headers) throws IOException { String[] arrayPartNos = partNos.split(OSDConstants.COMMA); for (String partNo : arrayPartNos) { - File partFile = new File(OSDUtils.getInstance().makeTempPath(path, objId, partNo)); + File partFile = new File(KsanUtils.makeTempPath(path, objId, partNo)); if (!partFile.delete()) { logger.error(OSDConstants.LOG_OSD_SERVER_FAILED_FILE_DELETE, partFile.getName()); @@ -985,34 +1159,64 @@ private void abortMultipart(String[] headers) throws IOException { // } private void sendHeader(Socket socket, String header) throws IOException { - byte[] buffer = header.getBytes(OSDConstants.CHARSET_UTF_8); - int size = buffer.length; + // byte[] buffer = header.getBytes(Constants.CHARSET_UTF_8); + // int size = buffer.length; - DataOutputStream so = new DataOutputStream(socket.getOutputStream()); + // DataOutputStream so = new DataOutputStream(socket.getOutputStream()); - so.writeInt(size); - so.write(buffer, 0, size); - so.flush(); + // so.writeInt(size); + // so.write(buffer, 0, size); + // so.flush(); + + byte[] buffer = header.getBytes(Constants.CHARSET_UTF_8); + String strLength = Integer.toString(buffer.length); + byte[] lengthBuffer = strLength.getBytes(Constants.CHARSET_UTF_8); + + byte length = (byte)lengthBuffer.length; + socket.getOutputStream().write(length); + socket.getOutputStream().write(lengthBuffer, 0, length); + + socket.getOutputStream().write(buffer, 0, buffer.length); + socket.getOutputStream().flush(); + logger.info("send header size : {}", buffer.length); } private void sendData(String ETag, long fileSize) throws IOException { String tail = OsdData.FILE + OsdData.DELIMITER + ETag + OsdData.DELIMITER + String.valueOf(fileSize); - byte[] buffer = tail.getBytes(OSDConstants.CHARSET_UTF_8); - int size = buffer.length; - - DataOutputStream so = new DataOutputStream(socket.getOutputStream()); + byte[] buffer = tail.getBytes(Constants.CHARSET_UTF_8); + String strLength = Integer.toString(buffer.length); + byte[] lengthBuffer = strLength.getBytes(Constants.CHARSET_UTF_8); + + byte length = (byte)lengthBuffer.length; + socket.getOutputStream().write(length); + socket.getOutputStream().write(lengthBuffer, 0, length); - so.writeInt(size); - so.write(buffer, 0, size); - so.flush(); + socket.getOutputStream().write(buffer, 0, buffer.length); + socket.getOutputStream().flush(); + logger.debug("sendData : {}", tail); + + // int size = buffer.length; + // logger.debug("sendData : {}", tail); + // DataOutputStream so = new DataOutputStream(socket.getOutputStream()); + // so.writeInt(size); + // so.write(buffer, 0, size); + // so.flush(); } private OsdData receiveData(Socket socket) throws IOException { - DataInputStream si = new DataInputStream(socket.getInputStream()); - int size = si.readInt(); - byte[] buffer = new byte[size]; - si.read(buffer, 0, size); - String result = new String(buffer, 0, size); + int length = socket.getInputStream().read(); + if (length == -1) { + logger.info("socket {} EOF ...", socket.getRemoteSocketAddress().toString()); + return null; + } + byte[] lengthBuffer = new byte[length]; + socket.getInputStream().read(lengthBuffer, 0, length); + String strLength = new String(lengthBuffer); + + length = Integer.parseInt(strLength); + byte[] buffer = new byte[length]; + socket.getInputStream().read(buffer, 0, length); + String result = new String(buffer, 0, length); String[] ArrayResult = result.split(OsdData.DELIMITER, -1); OsdData data = new OsdData(); @@ -1026,6 +1230,24 @@ private OsdData receiveData(Socket socket) throws IOException { } return null; + // DataInputStream si = new DataInputStream(socket.getInputStream()); + // int size = si.readInt(); + // byte[] buffer = new byte[size]; + // si.read(buffer, 0, size); + // String result = new String(buffer, 0, size); + // String[] ArrayResult = result.split(OsdData.DELIMITER, -1); + + // OsdData data = new OsdData(); + // switch (ArrayResult[0]) { + // case OsdData.FILE: + // data.setETag(ArrayResult[1]); + // data.setFileSize(Long.parseLong(ArrayResult[2])); + // return data; + // default: + // logger.error(OSDConstants.LOG_OSD_SERVER_UNKNOWN_DATA, ArrayResult[1]); + // } + + // return null; } private void retryRenameTo(File srcFile, File destFile) throws IOException { @@ -1066,6 +1288,7 @@ public static void startECThread() { } else { serviceEC = Executors.newSingleThreadScheduledExecutor(); } + logger.info("start ec thread, interval : {} ms", OSDConfig.getInstance().getECCheckInterval()); serviceEC.scheduleAtFixedRate(new DoECPriObject(), 1000, OSDConfig.getInstance().getECCheckInterval(), TimeUnit.MILLISECONDS); } diff --git a/core/src/com/pspace/ifs/ksan/osd/pom.xml b/core/src/com/pspace/ifs/ksan/osd/pom.xml index dbee64ea..3dc7bc87 100644 --- a/core/src/com/pspace/ifs/ksan/osd/pom.xml +++ b/core/src/com/pspace/ifs/ksan/osd/pom.xml @@ -79,6 +79,11 @@ commons-crypto 1.1.0 + + de.sfuhrm + openssl4j + 0.2.0 + diff --git a/core/src/com/pspace/ifs/ksan/osd/utils/OSDConstants.java b/core/src/com/pspace/ifs/ksan/osd/utils/OSDConstants.java index 08a5aba0..0a9b2af2 100644 --- a/core/src/com/pspace/ifs/ksan/osd/utils/OSDConstants.java +++ b/core/src/com/pspace/ifs/ksan/osd/utils/OSDConstants.java @@ -25,10 +25,10 @@ public final class OSDConstants { public static final String OSD_LOCAL_IP = "local_ip"; public static final String OSD_PORT = "port"; - public static final String OBJ_DIR = "obj"; - public static final String TEMP_DIR = "temp"; - public static final String TRASH_DIR = "trash"; - public static final String EC_DIR = "ec"; + // public static final String OBJ_DIR = "obj"; + // public static final String TEMP_DIR = "temp"; + // public static final String TRASH_DIR = "trash"; + // public static final String EC_DIR = "ec"; public static final String EC_FUNCTION = "ec"; public static final String EC_SCHEDULE_MINUTES = "ec_schedule_minutes"; @@ -45,11 +45,11 @@ public final class OSDConstants { public static final String PERFORMANCE_MODE = "performance_mode"; public static final String PERFORMANCE_MODE_NO_DISK = "NO_DISK"; - public static final String FILE_ATTRIBUTE_REPLICATION = "replication"; - public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID = "replica-diskid"; - public static final String FILE_ATTRUBUTE_REPLICATION_PRIMARY = "primary"; - public static final String FILE_ATTRIBUTE_REPLICATION_REPLICA = "replica"; - public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL = "null"; + // public static final String FILE_ATTRIBUTE_REPLICATION = "replication"; + // public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID = "replica-diskid"; + // public static final String FILE_ATTRUBUTE_REPLICATION_PRIMARY = "primary"; + // public static final String FILE_ATTRIBUTE_REPLICATION_REPLICA = "replica"; + // public static final String FILE_ATTRIBUTE_REPLICA_DISK_ID_NULL = "null"; public static final int HEADERSIZE = 1024 * 1024; public static final int MAXBUFSIZE = 524288; // 512 * 1024 @@ -169,7 +169,7 @@ public final class OSDConstants { public static final String LOG_OSD_SERVER_GET_PART_START = "getPart start ..."; public static final String LOG_OSD_SERVER_GET_PART_INFO = "path : {}, objId : {}, partNo : {}"; public static final String LOG_OSD_SERVER_GET_PART_END = "getPart end ... read total : {}"; - public static final String LOG_OSD_SERVER_GET_PART_SUCCESS_INFO = "get - success : path={}, objId={}, partNo={}"; + public static final String LOG_OSD_SERVER_GET_PART_SUCCESS_INFO = "getPart - success : path={}, objId={}, partNo={}"; public static final String LOG_OSD_SERVER_PART_START = "part start ..."; public static final String LOG_OSD_SERVER_PART_INFO = "path : {}, objId : {}, partNo : {}, length : {}, key : {}"; @@ -192,6 +192,15 @@ public final class OSDConstants { public static final String LOG_OSD_SERVER_ABORE_MULTIPART_END = "abortMultipart end ..."; public static final String LOG_OSD_SERVER_ABORE_MULTIPART_SUCCESS_INFO = "abortMultipart - success : path : {}, objId : {}, partNos : {}"; + public static final String LOG_OSD_SERVER_PUT_EC_PART_START = "put ec part start ..."; + public static final String LOG_OSD_SERVER_PUT_EC_PART_END = "put ec part end ..."; + + public static final String LOG_OSD_SERVER_GET_EC_PART_START = "get ec part start ..."; + public static final String LOG_OSD_SERVER_GET_EC_PART_END = "get ec part end ..."; + + public static final String LOG_OSD_SERVER_DELETE_EC_PART_START = "delete ec part start ..."; + public static final String LOG_OSD_SERVER_DELETE_EC_PART_END = "delete ec part end ..."; + // DoECPriObject public static final String LOG_DO_EC_PRI_OBJECT_START = "DoECPriObject start ..."; public static final String LOG_DO_EC_PRI_OBJECT_LOCAL_IP = "ip = {}"; @@ -200,9 +209,6 @@ public final class OSDConstants { public static final String LOG_DO_EC_PRI_OBJECT_FILE = "file : {}"; public static final String LOG_DO_EC_PRI_OBJECT_APPLY_MINUTES = "ec apply minutes : {}"; public static final String LOG_DO_EC_PRI_OBJECT_ENCODE_EC = "ENCODE EC : {}"; - public static final String DO_EC_PRI_OBJECT_ZFEC = "zfec -d "; - public static final String DO_EC_PRI_OBJECT_ZFEC_PREFIX_OPTION = " -p "; - public static final String DO_EC_PRI_OBJECT_ZFEC_TOTAL_NUMBER_OPTION = " -m 4 "; public static final String LOG_DO_EC_PRI_OBJECT_ZFEC_EXIT_CODE = "ENCODE exit code : {}"; public static final String LOG_DO_EC_PRI_OBJECT_ZFEC_COMMAND = "command : {}"; public static final String LOG_DO_EC_PRI_OBJECT_REPLICA_DISK_ID = "replica disk id : {}"; diff --git a/core/src/com/pspace/ifs/ksan/osd/utils/OSDUtils.java b/core/src/com/pspace/ifs/ksan/osd/utils/OSDUtils.java index c86ed976..93d326cd 100644 --- a/core/src/com/pspace/ifs/ksan/osd/utils/OSDUtils.java +++ b/core/src/com/pspace/ifs/ksan/osd/utils/OSDUtils.java @@ -122,68 +122,68 @@ private OSDUtils() { // return null; // } - private String makeDirectoryName(String objId) { - byte[] path = new byte[6]; - byte[] byteObjId = objId.getBytes(); + // private String makeDirectoryName(String objId) { + // byte[] path = new byte[6]; + // byte[] byteObjId = objId.getBytes(); - path[0] = OSDConstants.CHAR_SLASH; - int index = 1; + // path[0] = OSDConstants.CHAR_SLASH; + // int index = 1; - path[index++] = byteObjId[0]; - path[index++] = byteObjId[1]; - path[index++] = OSDConstants.CHAR_SLASH; - path[index++] = byteObjId[2]; - path[index] = byteObjId[3]; + // path[index++] = byteObjId[0]; + // path[index++] = byteObjId[1]; + // path[index++] = OSDConstants.CHAR_SLASH; + // path[index++] = byteObjId[2]; + // path[index] = byteObjId[3]; - return new String(path); - } + // return new String(path); + // } - public String makePath(String path, String fileName) { - String fullPath = path + OSDConstants.SLASH + OSDConstants.OBJ_DIR + makeDirectoryName(fileName) + OSDConstants.SLASH + fileName; - return fullPath; - } + // public String makePath(String path, String fileName) { + // String fullPath = path + OSDConstants.SLASH + OSDConstants.OBJ_DIR + makeDirectoryName(fileName) + OSDConstants.SLASH + fileName; + // return fullPath; + // } - public String makeObjPath(String path, String objId, String versionId) { - String fullPath = path + OSDConstants.SLASH + OSDConstants.OBJ_DIR + makeDirectoryName(objId) + OSDConstants.SLASH + objId + OSDConstants.UNDERSCORE + versionId; - return fullPath; - } + // public String makeObjPath(String path, String objId, String versionId) { + // String fullPath = path + OSDConstants.SLASH + OSDConstants.OBJ_DIR + makeDirectoryName(objId) + OSDConstants.SLASH + objId + OSDConstants.UNDERSCORE + versionId; + // return fullPath; + // } - public String makeTempPath(String path, String objId, String versionId) { - String uuid = UUID.randomUUID().toString(); - String fullPath = path + OSDConstants.SLASH + OSDConstants.TEMP_DIR + OSDConstants.SLASH + objId + OSDConstants.UNDERSCORE + uuid + OSDConstants.UNDERSCORE + versionId; - return fullPath; - } + // public String makeTempPath(String path, String objId, String versionId) { + // String uuid = UUID.randomUUID().toString(); + // String fullPath = path + OSDConstants.SLASH + OSDConstants.TEMP_DIR + OSDConstants.SLASH + objId + OSDConstants.UNDERSCORE + uuid + OSDConstants.UNDERSCORE + versionId; + // return fullPath; + // } - public String makeTempPartPath(String path, String objId, String partNumber) { - String fullPath = path + OSDConstants.SLASH + OSDConstants.TEMP_DIR + OSDConstants.SLASH + objId + OSDConstants.UNDERSCORE + partNumber; - return fullPath; - } + // public String makeTempPartPath(String path, String objId, String partNumber) { + // String fullPath = path + OSDConstants.SLASH + OSDConstants.TEMP_DIR + OSDConstants.SLASH + objId + OSDConstants.UNDERSCORE + partNumber; + // return fullPath; + // } - public String makeTrashPath(String path, String objId, String versionId) { - String uuid = UUID.randomUUID().toString(); - String fullPath = path + OSDConstants.SLASH + OSDConstants.TRASH_DIR + OSDConstants.SLASH + objId + OSDConstants.UNDERSCORE + versionId + uuid; - return fullPath; - } + // public String makeTrashPath(String path, String objId, String versionId) { + // String uuid = UUID.randomUUID().toString(); + // String fullPath = path + OSDConstants.SLASH + OSDConstants.TRASH_DIR + OSDConstants.SLASH + objId + OSDConstants.UNDERSCORE + versionId + uuid; + // return fullPath; + // } - public String makeCachePath(String path) { - String fullPath = OSDConfig.getInstance().getCacheDiskpath() + path; - return fullPath; - } + // public String makeCachePath(String path) { + // String fullPath = OSDConfig.getInstance().getCacheDiskpath() + path; + // return fullPath; + // } - public String makeECPath(String path, String objId, String versionId) { - String fullPath = path + OSDConstants.SLASH + OSDConstants.EC_DIR + makeDirectoryName(objId) + OSDConstants.SLASH + OSDConstants.POINT + objId + OSDConstants.UNDERSCORE + versionId; - return fullPath; - } + // public String makeECPath(String path, String objId, String versionId) { + // String fullPath = path + OSDConstants.SLASH + OSDConstants.EC_DIR + makeDirectoryName(objId) + OSDConstants.SLASH + OSDConstants.POINT + objId + OSDConstants.UNDERSCORE + versionId; + // return fullPath; + // } - public String makeECDirectory(String fileName, String ecPath) { - String fullPath = ecPath + makeDirectoryName(fileName); - return fullPath; - } + // public String makeECDirectory(String fileName, String ecPath) { + // String fullPath = ecPath + makeDirectoryName(fileName); + // return fullPath; + // } - public String makeECTempPath(String fileName, String ecPath) { - String fullPath = ecPath + makeDirectoryName(fileName) + OSDConstants.SLASH + OSDConstants.POINT + fileName; - return fullPath; - } + // public String makeECTempPath(String fileName, String ecPath) { + // String fullPath = ecPath + makeDirectoryName(fileName) + OSDConstants.SLASH + OSDConstants.POINT + fileName; + // return fullPath; + // } // public DISKPOOLLIST getDiskPoolList() { // DISKPOOLLIST diskpoolList = null; @@ -223,46 +223,50 @@ public String makeECTempPath(String fileName, String ecPath) { // } // } - public String getAttributeFileReplication(File file) { - UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); - ByteBuffer buf = null; - try { - buf = ByteBuffer.allocate(view.size(OSDConstants.FILE_ATTRIBUTE_REPLICATION)); - view.read(OSDConstants.FILE_ATTRIBUTE_REPLICATION, buf); - buf.flip(); - } catch (IOException e) { - logger.error(e.getMessage()); - } catch (IllegalArgumentException iae) { - logger.error(iae.getMessage()); - } catch (SecurityException e) { - logger.error(e.getMessage()); - } + // public String getAttributeFileReplication(File file) { + // UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); + // ByteBuffer buf = null; + // try { + // buf = ByteBuffer.allocate(view.size(OSDConstants.FILE_ATTRIBUTE_REPLICATION)); + // view.read(OSDConstants.FILE_ATTRIBUTE_REPLICATION, buf); + // buf.flip(); + // } catch (IOException e) { + // logger.error(e.getMessage()); + // } catch (IllegalArgumentException iae) { + // logger.error(iae.getMessage()); + // } catch (SecurityException e) { + // logger.error(e.getMessage()); + // } - return Charset.defaultCharset().decode(buf).toString(); - } + // return Charset.defaultCharset().decode(buf).toString(); + // } - public String getAttributeFileReplicaDiskID(File file) { - UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); - ByteBuffer buf = null; - try { - buf = ByteBuffer.allocate(view.size(OSDConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID)); - view.read(OSDConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID, buf); - buf.flip(); - } catch (IOException e) { - logger.error(e.getMessage()); - } catch (IllegalArgumentException iae) { - logger.error(iae.getMessage()); - } catch (SecurityException e) { - logger.error(e.getMessage()); - } + // public String getAttributeFileReplicaDiskID(File file) { + // UserDefinedFileAttributeView view = Files.getFileAttributeView(Paths.get(file.getPath()), UserDefinedFileAttributeView.class); + // ByteBuffer buf = null; + // try { + // buf = ByteBuffer.allocate(view.size(OSDConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID)); + // view.read(OSDConstants.FILE_ATTRIBUTE_REPLICA_DISK_ID, buf); + // buf.flip(); + // } catch (IOException e) { + // logger.error(e.getMessage()); + // } catch (IllegalArgumentException iae) { + // logger.error(iae.getMessage()); + // } catch (SecurityException e) { + // logger.error(e.getMessage()); + // } - return Charset.defaultCharset().decode(buf).toString(); - } + // return Charset.defaultCharset().decode(buf).toString(); + // } public static void sendHeader(Socket socket, String header) throws IOException { byte[] buffer = header.getBytes(OSDConstants.CHARSET_UTF_8); - byte length = (byte)buffer.length; + String strLength = Integer.toString(buffer.length); + byte[] lengthBuffer = strLength.getBytes(OSDConstants.CHARSET_UTF_8); + + byte length = (byte)lengthBuffer.length; socket.getOutputStream().write(length); + socket.getOutputStream().write(lengthBuffer, 0, length); socket.getOutputStream().write(buffer, 0, buffer.length); socket.getOutputStream().flush(); diff --git a/core/src/com/pspace/ifs/ksan/utils/cbalance/CBalanceMain.java b/core/src/com/pspace/ifs/ksan/utils/cbalance/CBalanceMain.java index 40e6110b..992df0ec 100644 --- a/core/src/com/pspace/ifs/ksan/utils/cbalance/CBalanceMain.java +++ b/core/src/com/pspace/ifs/ksan/utils/cbalance/CBalanceMain.java @@ -159,7 +159,7 @@ int parseArgs(String[] args){ return 0; - if (!bucketName.isEmpty() && (!key.isEmpty() || !objId.isEmpty()) && !SrcDiskName.isEmpty()) + if ((!bucketName.isEmpty() && (!key.isEmpty()) || (!objId.isEmpty()) && !SrcDiskName.isEmpty())) return 0; if (emptyDisk && !SrcDiskName.isEmpty()) diff --git a/docker/serivce/ksanlogmanager.service b/docker/serivce/ksanlogmanager.service new file mode 100644 index 00000000..c32364ef --- /dev/null +++ b/docker/serivce/ksanlogmanager.service @@ -0,0 +1,12 @@ +[Unit] +Description=KSAN Log Manager Service +Requires=docker.service +After=docker.service + +[Service] +Restart=always +ExecStart=/usr/bin/docker start -a ksan-log-manager +ExecStop=/usr/bin/docker stop -t 2 ksan-log-manager + +[Install] +WantedBy=multi-user.target \ No newline at end of file diff --git a/docker/serivce/ksanreplicationmanager.service b/docker/serivce/ksanreplicationmanager.service new file mode 100644 index 00000000..bca17be7 --- /dev/null +++ b/docker/serivce/ksanreplicationmanager.service @@ -0,0 +1,12 @@ +[Unit] +Description=KSAN Replication Manager Service +Requires=docker.service +After=docker.service + +[Service] +Restart=always +ExecStart=/usr/bin/docker start -a ksan-replication-manager +ExecStop=/usr/bin/docker stop -t 2 ksan-replication-manager + +[Install] +WantedBy=multi-user.target \ No newline at end of file diff --git a/portal/Data/PortalData/Configs/KeyAndDefaultValueAttribute.cs b/portal/Data/PortalData/Configs/KeyAndDefaultValueAttribute.cs index 693aaed8..81d7c216 100644 --- a/portal/Data/PortalData/Configs/KeyAndDefaultValueAttribute.cs +++ b/portal/Data/PortalData/Configs/KeyAndDefaultValueAttribute.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumDiskControl.cs b/portal/Data/PortalData/Enums/EnumDiskControl.cs index fe249d0f..11a375f5 100644 --- a/portal/Data/PortalData/Enums/EnumDiskControl.cs +++ b/portal/Data/PortalData/Enums/EnumDiskControl.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumDiskPoolReplicaType.cs b/portal/Data/PortalData/Enums/EnumDiskPoolReplicaType.cs index 70c8bda9..c3976546 100644 --- a/portal/Data/PortalData/Enums/EnumDiskPoolReplicaType.cs +++ b/portal/Data/PortalData/Enums/EnumDiskPoolReplicaType.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -27,6 +27,9 @@ public enum EnumDiskPoolReplicaType OnePlusOne, /// 1+2 [Display(Name = "UL_DISKPOOL_REPLICA_TYPE_ONE_PULS_TWO", Description = "UL_DISKPOOL_REPLICA_TYPE_ONE_PULS_TWO", GroupName = "UL_DISKPOOL_REPLICA_TYPE", ShortName = "UL_DISKPOOL_REPLICA_TYPE_ONE_PULS_TWO", Prompt = "UL_DISKPOOL_REPLICA_TYPE_ONE_PULS_TWO", ResourceType = typeof(Resource))] - OnePlusTwo + OnePlusTwo, + /// Erasure Code + [Display(Name = "UL_DISKPOOL_REPLICA_TYPE_EC", Description = "UL_DISKPOOL_REPLICA_TYPE_EC", GroupName = "UL_DISKPOOL_REPLICA_TYPE", ShortName = "UL_DISKPOOL_REPLICA_TYPE_EC", Prompt = "UL_DISKPOOL_REPLICA_TYPE_EC", ResourceType = typeof(Resource))] + ErasureCode } } \ No newline at end of file diff --git a/portal/Data/PortalData/Enums/EnumDiskPoolType.cs b/portal/Data/PortalData/Enums/EnumDiskPoolType.cs index 63705860..67d2bd4c 100644 --- a/portal/Data/PortalData/Enums/EnumDiskPoolType.cs +++ b/portal/Data/PortalData/Enums/EnumDiskPoolType.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumDiskRwMode.cs b/portal/Data/PortalData/Enums/EnumDiskRwMode.cs index a5928b4f..e62db1ca 100644 --- a/portal/Data/PortalData/Enums/EnumDiskRwMode.cs +++ b/portal/Data/PortalData/Enums/EnumDiskRwMode.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumDiskState.cs b/portal/Data/PortalData/Enums/EnumDiskState.cs index 4bc0c534..1f351cec 100644 --- a/portal/Data/PortalData/Enums/EnumDiskState.cs +++ b/portal/Data/PortalData/Enums/EnumDiskState.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumHaAction.cs b/portal/Data/PortalData/Enums/EnumHaAction.cs index 1310bfb2..7968c60f 100644 --- a/portal/Data/PortalData/Enums/EnumHaAction.cs +++ b/portal/Data/PortalData/Enums/EnumHaAction.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumHaProxyBalance.cs b/portal/Data/PortalData/Enums/EnumHaProxyBalance.cs index 80a237ec..5ab8c1e0 100644 --- a/portal/Data/PortalData/Enums/EnumHaProxyBalance.cs +++ b/portal/Data/PortalData/Enums/EnumHaProxyBalance.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumHaProxyListenMode.cs b/portal/Data/PortalData/Enums/EnumHaProxyListenMode.cs index 311cfeea..f940a742 100644 --- a/portal/Data/PortalData/Enums/EnumHaProxyListenMode.cs +++ b/portal/Data/PortalData/Enums/EnumHaProxyListenMode.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumLogLevel.cs b/portal/Data/PortalData/Enums/EnumLogLevel.cs index c96d4cf8..3703ad59 100644 --- a/portal/Data/PortalData/Enums/EnumLogLevel.cs +++ b/portal/Data/PortalData/Enums/EnumLogLevel.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumNetworkLinkState.cs b/portal/Data/PortalData/Enums/EnumNetworkLinkState.cs index 78de7a0c..96b9e451 100644 --- a/portal/Data/PortalData/Enums/EnumNetworkLinkState.cs +++ b/portal/Data/PortalData/Enums/EnumNetworkLinkState.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumOnOff.cs b/portal/Data/PortalData/Enums/EnumOnOff.cs index 58bf08de..dea243e7 100644 --- a/portal/Data/PortalData/Enums/EnumOnOff.cs +++ b/portal/Data/PortalData/Enums/EnumOnOff.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumServerState.cs b/portal/Data/PortalData/Enums/EnumServerState.cs index 2e6a5072..61626870 100644 --- a/portal/Data/PortalData/Enums/EnumServerState.cs +++ b/portal/Data/PortalData/Enums/EnumServerState.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumServiceControl.cs b/portal/Data/PortalData/Enums/EnumServiceControl.cs index d69c6861..71673bcf 100644 --- a/portal/Data/PortalData/Enums/EnumServiceControl.cs +++ b/portal/Data/PortalData/Enums/EnumServiceControl.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumServiceEventType.cs b/portal/Data/PortalData/Enums/EnumServiceEventType.cs index 08f4bcbd..98e27eb9 100644 --- a/portal/Data/PortalData/Enums/EnumServiceEventType.cs +++ b/portal/Data/PortalData/Enums/EnumServiceEventType.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,12 +10,11 @@ */ using Newtonsoft.Json; using Newtonsoft.Json.Converters; -using Newtonsoft.Json.Serialization; namespace PortalData.Enums { /// 서비스 이벤트 - [JsonConverter(typeof(StringEnumConverter), typeof(CamelCaseNamingStrategy))] + [JsonConverter(typeof(StringEnumConverter))] public enum EnumServiceEventType { /// Error diff --git a/portal/Data/PortalData/Enums/EnumServiceGroupMode.cs b/portal/Data/PortalData/Enums/EnumServiceGroupMode.cs index 84efc308..3ab5d5cf 100644 --- a/portal/Data/PortalData/Enums/EnumServiceGroupMode.cs +++ b/portal/Data/PortalData/Enums/EnumServiceGroupMode.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumServiceState.cs b/portal/Data/PortalData/Enums/EnumServiceState.cs index 96c2d878..f9e4adb7 100644 --- a/portal/Data/PortalData/Enums/EnumServiceState.cs +++ b/portal/Data/PortalData/Enums/EnumServiceState.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumServiceType.cs b/portal/Data/PortalData/Enums/EnumServiceType.cs index 2def36b5..3b5b1eb1 100644 --- a/portal/Data/PortalData/Enums/EnumServiceType.cs +++ b/portal/Data/PortalData/Enums/EnumServiceType.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,12 +10,11 @@ */ using Newtonsoft.Json; using Newtonsoft.Json.Converters; -using Newtonsoft.Json.Serialization; namespace PortalData.Enums { /// 서비스 타입 - [JsonConverter(typeof(StringEnumConverter), typeof(CamelCaseNamingStrategy))] + [JsonConverter(typeof(StringEnumConverter))] public enum EnumServiceType { /// Unknown @@ -35,20 +34,18 @@ public enum EnumServiceType /// PortalBridge ksanPortalBridge, /// Agent - KsanAgent, + ksanAgent, /// OSD ksanOSD, /// GW ksanGW, /// Recovery ksanRecovery, - /// Lifecycle - ksanLifecycle, - /// Replication - ksanReplication, - /// LogManager - ksanLogManager, - /// Metering - ksanMetering + /// Lifecycle Manager + ksanLifecycleManager, + /// Replication Manager + ksanReplicationManager, + /// Log Manager + ksanLogManager } } \ No newline at end of file diff --git a/portal/Data/PortalData/Enums/EnumUserStatus.cs b/portal/Data/PortalData/Enums/EnumUserStatus.cs index 548ae8df..152b20b8 100644 --- a/portal/Data/PortalData/Enums/EnumUserStatus.cs +++ b/portal/Data/PortalData/Enums/EnumUserStatus.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Enums/EnumYesNo.cs b/portal/Data/PortalData/Enums/EnumYesNo.cs index 76693217..3119fe88 100644 --- a/portal/Data/PortalData/Enums/EnumYesNo.cs +++ b/portal/Data/PortalData/Enums/EnumYesNo.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestAddClaimToRole.cs b/portal/Data/PortalData/Requests/Accounts/RequestAddClaimToRole.cs index 9c76ec25..808339d8 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestAddClaimToRole.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestAddClaimToRole.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestAddClaimToUser.cs b/portal/Data/PortalData/Requests/Accounts/RequestAddClaimToUser.cs index 58c0e651..f47b8081 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestAddClaimToUser.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestAddClaimToUser.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestAddRoleToUser.cs b/portal/Data/PortalData/Requests/Accounts/RequestAddRoleToUser.cs index b683cb45..8b9f3fb7 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestAddRoleToUser.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestAddRoleToUser.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestAddUserToRole.cs b/portal/Data/PortalData/Requests/Accounts/RequestAddUserToRole.cs index 056ccc58..2aeca4c7 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestAddUserToRole.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestAddUserToRole.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestApiKey.cs b/portal/Data/PortalData/Requests/Accounts/RequestApiKey.cs index f4659606..a66f6df1 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestApiKey.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestApiKey.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestApiKeyEx.cs b/portal/Data/PortalData/Requests/Accounts/RequestApiKeyEx.cs index ebfd80de..dbcd6bae 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestApiKeyEx.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestApiKeyEx.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestChangePassword.cs b/portal/Data/PortalData/Requests/Accounts/RequestChangePassword.cs index d7467127..2e1a86d4 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestChangePassword.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestChangePassword.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestConfirmEmail.cs b/portal/Data/PortalData/Requests/Accounts/RequestConfirmEmail.cs index 93511351..5cb298b1 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestConfirmEmail.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestConfirmEmail.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestExternalAccountVerify.cs b/portal/Data/PortalData/Requests/Accounts/RequestExternalAccountVerify.cs index e02f4ad8..167431f2 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestExternalAccountVerify.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestExternalAccountVerify.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestExternalLogin.cs b/portal/Data/PortalData/Requests/Accounts/RequestExternalLogin.cs index efca0c25..ab14ce37 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestExternalLogin.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestExternalLogin.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestExternalRegister.cs b/portal/Data/PortalData/Requests/Accounts/RequestExternalRegister.cs index fbb177b3..d4696168 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestExternalRegister.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestExternalRegister.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestForgetPassword.cs b/portal/Data/PortalData/Requests/Accounts/RequestForgetPassword.cs index 42725aa7..965a28c7 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestForgetPassword.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestForgetPassword.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestLogin.cs b/portal/Data/PortalData/Requests/Accounts/RequestLogin.cs index 485d9659..d325161a 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestLogin.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestLogin.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestRegister.cs b/portal/Data/PortalData/Requests/Accounts/RequestRegister.cs index fe0ddbc9..e7ceb3e7 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestRegister.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestRegister.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestResetPassword.cs b/portal/Data/PortalData/Requests/Accounts/RequestResetPassword.cs index 566e8306..81dca642 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestResetPassword.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestResetPassword.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestRole.cs b/portal/Data/PortalData/Requests/Accounts/RequestRole.cs index 2459adf6..fcf93a6f 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestRole.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestRole.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestRoleClaim.cs b/portal/Data/PortalData/Requests/Accounts/RequestRoleClaim.cs index e082afb7..01352fe2 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestRoleClaim.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestRoleClaim.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestSnasUserRegister.cs b/portal/Data/PortalData/Requests/Accounts/RequestSnasUserRegister.cs index 50dfe0f1..63540d6d 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestSnasUserRegister.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestSnasUserRegister.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestUpdate.cs b/portal/Data/PortalData/Requests/Accounts/RequestUpdate.cs index 2f993fd6..8c03cafb 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestUpdate.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestUpdate.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestUserChangePassword.cs b/portal/Data/PortalData/Requests/Accounts/RequestUserChangePassword.cs index 4292de7e..ddb88d29 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestUserChangePassword.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestUserChangePassword.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestUserRegist.cs b/portal/Data/PortalData/Requests/Accounts/RequestUserRegist.cs index 8e37d09e..c8b74213 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestUserRegist.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestUserRegist.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Accounts/RequestUserUpdate.cs b/portal/Data/PortalData/Requests/Accounts/RequestUserUpdate.cs index a579f8f1..dd08ada7 100644 --- a/portal/Data/PortalData/Requests/Accounts/RequestUserUpdate.cs +++ b/portal/Data/PortalData/Requests/Accounts/RequestUserUpdate.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Agent/RequestAgentInitialize.cs b/portal/Data/PortalData/Requests/Agent/RequestAgentInitialize.cs index 08acf17d..3437f284 100644 --- a/portal/Data/PortalData/Requests/Agent/RequestAgentInitialize.cs +++ b/portal/Data/PortalData/Requests/Agent/RequestAgentInitialize.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Configs/RequestServiceConfig.cs b/portal/Data/PortalData/Requests/Configs/RequestServiceConfig.cs index e9437bcd..1857799c 100644 --- a/portal/Data/PortalData/Requests/Configs/RequestServiceConfig.cs +++ b/portal/Data/PortalData/Requests/Configs/RequestServiceConfig.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Disks/RequestDisk.cs b/portal/Data/PortalData/Requests/Disks/RequestDisk.cs index 18c5a7dd..9b212357 100644 --- a/portal/Data/PortalData/Requests/Disks/RequestDisk.cs +++ b/portal/Data/PortalData/Requests/Disks/RequestDisk.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -19,6 +19,9 @@ namespace PortalData.Requests.Disks /// 디스크 등록/수정 요청 클래스 public class RequestDisk : CommonRequestData { + /// 서버 아이디 + public string ServerId { get; set; } + /// 디스크 풀 아이디 public string DiskPoolId { get; set; } diff --git a/portal/Data/PortalData/Requests/Disks/RequestDiskAlreadyMount.cs b/portal/Data/PortalData/Requests/Disks/RequestDiskAlreadyMount.cs index 45c384b4..3f3a8af8 100644 --- a/portal/Data/PortalData/Requests/Disks/RequestDiskAlreadyMount.cs +++ b/portal/Data/PortalData/Requests/Disks/RequestDiskAlreadyMount.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Disks/RequestDiskPool.cs b/portal/Data/PortalData/Requests/Disks/RequestDiskPool.cs index a83fc4b7..b199b96c 100644 --- a/portal/Data/PortalData/Requests/Disks/RequestDiskPool.cs +++ b/portal/Data/PortalData/Requests/Disks/RequestDiskPool.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Disks/RequestDiskThreshold.cs b/portal/Data/PortalData/Requests/Disks/RequestDiskThreshold.cs index 7275aa87..276b460d 100644 --- a/portal/Data/PortalData/Requests/Disks/RequestDiskThreshold.cs +++ b/portal/Data/PortalData/Requests/Disks/RequestDiskThreshold.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Disks/RequestDiskUsage.cs b/portal/Data/PortalData/Requests/Disks/RequestDiskUsage.cs index 933a0053..3c2c6ff0 100644 --- a/portal/Data/PortalData/Requests/Disks/RequestDiskUsage.cs +++ b/portal/Data/PortalData/Requests/Disks/RequestDiskUsage.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Disks/RequestDisks.cs b/portal/Data/PortalData/Requests/Disks/RequestDisks.cs index a65bf82d..1d19a24a 100644 --- a/portal/Data/PortalData/Requests/Disks/RequestDisks.cs +++ b/portal/Data/PortalData/Requests/Disks/RequestDisks.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Disks/RequestToWriteDiskId.cs b/portal/Data/PortalData/Requests/Disks/RequestToWriteDiskId.cs index 95f4f8b7..4fb8de32 100644 --- a/portal/Data/PortalData/Requests/Disks/RequestToWriteDiskId.cs +++ b/portal/Data/PortalData/Requests/Disks/RequestToWriteDiskId.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Ksan/RequestKsanUser.cs b/portal/Data/PortalData/Requests/Ksan/RequestKsanUser.cs index d3a08bcf..9a131ac0 100644 --- a/portal/Data/PortalData/Requests/Ksan/RequestKsanUser.cs +++ b/portal/Data/PortalData/Requests/Ksan/RequestKsanUser.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Ksan/RequestKsanUserUpdate.cs b/portal/Data/PortalData/Requests/Ksan/RequestKsanUserUpdate.cs index ac26f9eb..adabda14 100644 --- a/portal/Data/PortalData/Requests/Ksan/RequestKsanUserUpdate.cs +++ b/portal/Data/PortalData/Requests/Ksan/RequestKsanUserUpdate.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Ksan/RequestStorageClass.cs b/portal/Data/PortalData/Requests/Ksan/RequestStorageClass.cs index 547de456..012790b5 100644 --- a/portal/Data/PortalData/Requests/Ksan/RequestStorageClass.cs +++ b/portal/Data/PortalData/Requests/Ksan/RequestStorageClass.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Networks/RequestIsNetworkInterfaceNameExist.cs b/portal/Data/PortalData/Requests/Networks/RequestIsNetworkInterfaceNameExist.cs index 227f3218..ce00449c 100644 --- a/portal/Data/PortalData/Requests/Networks/RequestIsNetworkInterfaceNameExist.cs +++ b/portal/Data/PortalData/Requests/Networks/RequestIsNetworkInterfaceNameExist.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Networks/RequestIsNetworkInterfaceVlanExist.cs b/portal/Data/PortalData/Requests/Networks/RequestIsNetworkInterfaceVlanExist.cs index a9c0aad8..d63cd1ca 100644 --- a/portal/Data/PortalData/Requests/Networks/RequestIsNetworkInterfaceVlanExist.cs +++ b/portal/Data/PortalData/Requests/Networks/RequestIsNetworkInterfaceVlanExist.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterface.cs b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterface.cs index 516cb90e..687dc967 100644 --- a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterface.cs +++ b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterface.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceLinkState.cs b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceLinkState.cs index 189d06c1..36dabf8f 100644 --- a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceLinkState.cs +++ b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceLinkState.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceUsage.cs b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceUsage.cs index 49812847..289309d0 100644 --- a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceUsage.cs +++ b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceUsage.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceVlan.cs b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceVlan.cs index 855fd82d..cc88a326 100644 --- a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceVlan.cs +++ b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceVlan.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceVlanUsage.cs b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceVlanUsage.cs index 76be4389..cc4e2fca 100644 --- a/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceVlanUsage.cs +++ b/portal/Data/PortalData/Requests/Networks/RequestNetworkInterfaceVlanUsage.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Region/RequestRegion.cs b/portal/Data/PortalData/Requests/Region/RequestRegion.cs index 8e19b6f9..275e2e4e 100644 --- a/portal/Data/PortalData/Requests/Region/RequestRegion.cs +++ b/portal/Data/PortalData/Requests/Region/RequestRegion.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Region/RequestRegionSync.cs b/portal/Data/PortalData/Requests/Region/RequestRegionSync.cs index 0d0f7fad..a4e4a2e6 100644 --- a/portal/Data/PortalData/Requests/Region/RequestRegionSync.cs +++ b/portal/Data/PortalData/Requests/Region/RequestRegionSync.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Servers/RequestServer.cs b/portal/Data/PortalData/Requests/Servers/RequestServer.cs index 1b77d080..b9821cf8 100644 --- a/portal/Data/PortalData/Requests/Servers/RequestServer.cs +++ b/portal/Data/PortalData/Requests/Servers/RequestServer.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Servers/RequestServerInitialize.cs b/portal/Data/PortalData/Requests/Servers/RequestServerInitialize.cs index 9a28e2d3..64e3f857 100644 --- a/portal/Data/PortalData/Requests/Servers/RequestServerInitialize.cs +++ b/portal/Data/PortalData/Requests/Servers/RequestServerInitialize.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Servers/RequestServerState.cs b/portal/Data/PortalData/Requests/Servers/RequestServerState.cs index d35f1c29..f7ac4065 100644 --- a/portal/Data/PortalData/Requests/Servers/RequestServerState.cs +++ b/portal/Data/PortalData/Requests/Servers/RequestServerState.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Servers/RequestServerUsage.cs b/portal/Data/PortalData/Requests/Servers/RequestServerUsage.cs index 282573e1..93effa2e 100644 --- a/portal/Data/PortalData/Requests/Servers/RequestServerUsage.cs +++ b/portal/Data/PortalData/Requests/Servers/RequestServerUsage.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestIsServiceGroupNameExist.cs b/portal/Data/PortalData/Requests/Services/RequestIsServiceGroupNameExist.cs index 04b8cd30..26a8b00a 100644 --- a/portal/Data/PortalData/Requests/Services/RequestIsServiceGroupNameExist.cs +++ b/portal/Data/PortalData/Requests/Services/RequestIsServiceGroupNameExist.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestService.cs b/portal/Data/PortalData/Requests/Services/RequestService.cs index 64feab31..1a08a065 100644 --- a/portal/Data/PortalData/Requests/Services/RequestService.cs +++ b/portal/Data/PortalData/Requests/Services/RequestService.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestServiceControl.cs b/portal/Data/PortalData/Requests/Services/RequestServiceControl.cs index 69be15a5..807ae890 100644 --- a/portal/Data/PortalData/Requests/Services/RequestServiceControl.cs +++ b/portal/Data/PortalData/Requests/Services/RequestServiceControl.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestServiceEvent.cs b/portal/Data/PortalData/Requests/Services/RequestServiceEvent.cs index 702dc009..54b43875 100644 --- a/portal/Data/PortalData/Requests/Services/RequestServiceEvent.cs +++ b/portal/Data/PortalData/Requests/Services/RequestServiceEvent.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestServiceGroup.cs b/portal/Data/PortalData/Requests/Services/RequestServiceGroup.cs index 52a613ef..dab97677 100644 --- a/portal/Data/PortalData/Requests/Services/RequestServiceGroup.cs +++ b/portal/Data/PortalData/Requests/Services/RequestServiceGroup.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestServiceHaAction.cs b/portal/Data/PortalData/Requests/Services/RequestServiceHaAction.cs index f150721b..48341f10 100644 --- a/portal/Data/PortalData/Requests/Services/RequestServiceHaAction.cs +++ b/portal/Data/PortalData/Requests/Services/RequestServiceHaAction.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestServiceLoadConfig.cs b/portal/Data/PortalData/Requests/Services/RequestServiceLoadConfig.cs index aec97f2d..e490d8ec 100644 --- a/portal/Data/PortalData/Requests/Services/RequestServiceLoadConfig.cs +++ b/portal/Data/PortalData/Requests/Services/RequestServiceLoadConfig.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestServiceSaveConfig.cs b/portal/Data/PortalData/Requests/Services/RequestServiceSaveConfig.cs index b1b2c9ca..f993dd33 100644 --- a/portal/Data/PortalData/Requests/Services/RequestServiceSaveConfig.cs +++ b/portal/Data/PortalData/Requests/Services/RequestServiceSaveConfig.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestServiceState.cs b/portal/Data/PortalData/Requests/Services/RequestServiceState.cs index d777f0a6..62f432d0 100644 --- a/portal/Data/PortalData/Requests/Services/RequestServiceState.cs +++ b/portal/Data/PortalData/Requests/Services/RequestServiceState.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Requests/Services/RequestServiceUsage.cs b/portal/Data/PortalData/Requests/Services/RequestServiceUsage.cs index f1fa19b7..bae6d70a 100644 --- a/portal/Data/PortalData/Requests/Services/RequestServiceUsage.cs +++ b/portal/Data/PortalData/Requests/Services/RequestServiceUsage.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/ResponseData.cs b/portal/Data/PortalData/ResponseData.cs index c71a6b48..82c092e9 100644 --- a/portal/Data/PortalData/ResponseData.cs +++ b/portal/Data/PortalData/ResponseData.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/ResponseDataSimple.cs b/portal/Data/PortalData/ResponseDataSimple.cs index 5a1bebae..5f485f0f 100644 --- a/portal/Data/PortalData/ResponseDataSimple.cs +++ b/portal/Data/PortalData/ResponseDataSimple.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/ResponseDataWithCreatedBy.cs b/portal/Data/PortalData/ResponseDataWithCreatedBy.cs index edf2c7fd..f0d57e04 100644 --- a/portal/Data/PortalData/ResponseDataWithCreatedBy.cs +++ b/portal/Data/PortalData/ResponseDataWithCreatedBy.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/ResponseDataWithModifiedBy.cs b/portal/Data/PortalData/ResponseDataWithModifiedBy.cs index 37016af9..a7bcc943 100644 --- a/portal/Data/PortalData/ResponseDataWithModifiedBy.cs +++ b/portal/Data/PortalData/ResponseDataWithModifiedBy.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/ResponseList.cs b/portal/Data/PortalData/ResponseList.cs index c9d1d703..ae7c36a3 100644 --- a/portal/Data/PortalData/ResponseList.cs +++ b/portal/Data/PortalData/ResponseList.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/ResponseMqData.cs b/portal/Data/PortalData/ResponseMqData.cs index 225df2d6..363934a3 100644 --- a/portal/Data/PortalData/ResponseMqData.cs +++ b/portal/Data/PortalData/ResponseMqData.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Accounts/ResponseApiKey.cs b/portal/Data/PortalData/Responses/Accounts/ResponseApiKey.cs index fee93644..57664362 100644 --- a/portal/Data/PortalData/Responses/Accounts/ResponseApiKey.cs +++ b/portal/Data/PortalData/Responses/Accounts/ResponseApiKey.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Accounts/ResponseClaim.cs b/portal/Data/PortalData/Responses/Accounts/ResponseClaim.cs index e1df508c..e6a05fd5 100644 --- a/portal/Data/PortalData/Responses/Accounts/ResponseClaim.cs +++ b/portal/Data/PortalData/Responses/Accounts/ResponseClaim.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Accounts/ResponseClaimGroup.cs b/portal/Data/PortalData/Responses/Accounts/ResponseClaimGroup.cs index 85142857..bb131297 100644 --- a/portal/Data/PortalData/Responses/Accounts/ResponseClaimGroup.cs +++ b/portal/Data/PortalData/Responses/Accounts/ResponseClaimGroup.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Accounts/ResponseLogin.cs b/portal/Data/PortalData/Responses/Accounts/ResponseLogin.cs index 445749b2..f725d475 100644 --- a/portal/Data/PortalData/Responses/Accounts/ResponseLogin.cs +++ b/portal/Data/PortalData/Responses/Accounts/ResponseLogin.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Accounts/ResponseRole.cs b/portal/Data/PortalData/Responses/Accounts/ResponseRole.cs index 9a82a821..6d35d2ba 100644 --- a/portal/Data/PortalData/Responses/Accounts/ResponseRole.cs +++ b/portal/Data/PortalData/Responses/Accounts/ResponseRole.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Accounts/ResponseUser.cs b/portal/Data/PortalData/Responses/Accounts/ResponseUser.cs index 36fe42ca..7b8524ba 100644 --- a/portal/Data/PortalData/Responses/Accounts/ResponseUser.cs +++ b/portal/Data/PortalData/Responses/Accounts/ResponseUser.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Accounts/ResponseUserWithRoles.cs b/portal/Data/PortalData/Responses/Accounts/ResponseUserWithRoles.cs index 01d301c2..880c92b1 100644 --- a/portal/Data/PortalData/Responses/Accounts/ResponseUserWithRoles.cs +++ b/portal/Data/PortalData/Responses/Accounts/ResponseUserWithRoles.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Common/ResponseAllowConnectionIp.cs b/portal/Data/PortalData/Responses/Common/ResponseAllowConnectionIp.cs index b71d6f3f..0c31150a 100644 --- a/portal/Data/PortalData/Responses/Common/ResponseAllowConnectionIp.cs +++ b/portal/Data/PortalData/Responses/Common/ResponseAllowConnectionIp.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Common/ResponseConfig.cs b/portal/Data/PortalData/Responses/Common/ResponseConfig.cs index fad914df..b8d90e97 100644 --- a/portal/Data/PortalData/Responses/Common/ResponseConfig.cs +++ b/portal/Data/PortalData/Responses/Common/ResponseConfig.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Common/ResponseSiteConfig.cs b/portal/Data/PortalData/Responses/Common/ResponseSiteConfig.cs index 31d6fe7c..e7fb9a8b 100644 --- a/portal/Data/PortalData/Responses/Common/ResponseSiteConfig.cs +++ b/portal/Data/PortalData/Responses/Common/ResponseSiteConfig.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Common/ResponseUpload.cs b/portal/Data/PortalData/Responses/Common/ResponseUpload.cs index 5b5f1e25..2218ead5 100644 --- a/portal/Data/PortalData/Responses/Common/ResponseUpload.cs +++ b/portal/Data/PortalData/Responses/Common/ResponseUpload.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Configs/ResponseServiceConfig.cs b/portal/Data/PortalData/Responses/Configs/ResponseServiceConfig.cs index d74a6041..bf76e857 100644 --- a/portal/Data/PortalData/Responses/Configs/ResponseServiceConfig.cs +++ b/portal/Data/PortalData/Responses/Configs/ResponseServiceConfig.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Configs/ResponseUpdateConfig.cs b/portal/Data/PortalData/Responses/Configs/ResponseUpdateConfig.cs index e031cff6..234124b9 100644 --- a/portal/Data/PortalData/Responses/Configs/ResponseUpdateConfig.cs +++ b/portal/Data/PortalData/Responses/Configs/ResponseUpdateConfig.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDisk.cs b/portal/Data/PortalData/Responses/Disks/ResponseDisk.cs index 7006bf84..456e9d13 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDisk.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDisk.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -21,12 +21,15 @@ public class ResponseDisk /// 서버 아이디 public string ServerId { get; set; } - /// 디스크 풀 이름 - public string DiskPoolName { get; set; } + /// 서버 이름 + public string ServerName { get; set; } /// 디스크 풀 아이디 public string DiskPoolId { get; set; } + /// 디스크 풀 이름 + public string DiskPoolName { get; set; } + /// 디스크 이름 public string Name { get; set; } diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDiskId.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskId.cs index e4b48a0c..32ee8580 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDiskId.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskId.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDiskPool.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskPool.cs index fff36209..8dce95dc 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDiskPool.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskPool.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -28,5 +28,8 @@ public class ResponseDiskPool : ResponseDataWithModifiedBy /// 복제 타입 public EnumDiskPoolReplicaType ReplicationType { get; set; } + + /// EC 설정 정보 + public ResponseDiskPoolEC EC { get; set; } = new ResponseDiskPoolEC(); } } \ No newline at end of file diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolDetails.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolDetails.cs index 4dc408b1..9adcd14b 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolDetails.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolDetails.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Portal/src/app/shared/models/response-data-simple.model.ts b/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolEC.cs similarity index 67% rename from portal/Portal/src/app/shared/models/response-data-simple.model.ts rename to portal/Data/PortalData/Responses/Disks/ResponseDiskPoolEC.cs index 8fe4725e..e3a60ef7 100644 --- a/portal/Portal/src/app/shared/models/response-data-simple.model.ts +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolEC.cs @@ -2,26 +2,21 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ -import {CommonResponseDataWithAuth} from './commondata/common-response-data-with-auth.model'; - -/** - * 상세 데이터가 없는 데이터 응답 클래스 - */ -export class ResponseData extends CommonResponseDataWithAuth { - - /** - * 생성자 - */ - constructor() { - super(); +namespace PortalData.Responses.Disks +{ + /// 디스크풀 DC 설정 정보 + public class ResponseDiskPoolEC + { + /// 원본 파일 등분 갯수 + public int M { get; set; } + /// 인코딩 갯수 + public int K { get; set; } } -} - - +} \ No newline at end of file diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolWithDisks.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolWithDisks.cs index e2349e54..a76046aa 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolWithDisks.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskPoolWithDisks.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDiskRwMode.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskRwMode.cs index db104d1a..67a41773 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDiskRwMode.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskRwMode.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDiskState.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskState.cs index 87f3ec2e..f3a80bc4 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDiskState.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskState.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Disks/RequestDiskThreshold.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskThreshold.cs similarity index 96% rename from portal/Data/PortalData/Responses/Disks/RequestDiskThreshold.cs rename to portal/Data/PortalData/Responses/Disks/ResponseDiskThreshold.cs index 8d717340..e70dfb27 100644 --- a/portal/Data/PortalData/Responses/Disks/RequestDiskThreshold.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskThreshold.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDiskWithDiskPool.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskWithDiskPool.cs index 9f709989..0a1e4960 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDiskWithDiskPool.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskWithDiskPool.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDiskWithServer.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskWithServer.cs index 974410b3..973953f3 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDiskWithServer.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskWithServer.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Disks/ResponseDiskWithServerAndNetwork.cs b/portal/Data/PortalData/Responses/Disks/ResponseDiskWithServerAndNetwork.cs index fdf0d352..f4963121 100644 --- a/portal/Data/PortalData/Responses/Disks/ResponseDiskWithServerAndNetwork.cs +++ b/portal/Data/PortalData/Responses/Disks/ResponseDiskWithServerAndNetwork.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Ksan/ResponseKsanUser.cs b/portal/Data/PortalData/Responses/Ksan/ResponseKsanUser.cs index ce3dfd3b..13e5c6bc 100644 --- a/portal/Data/PortalData/Responses/Ksan/ResponseKsanUser.cs +++ b/portal/Data/PortalData/Responses/Ksan/ResponseKsanUser.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Ksan/ResponseStorageClass.cs b/portal/Data/PortalData/Responses/Ksan/ResponseStorageClass.cs index 78ce8d14..3615df05 100644 --- a/portal/Data/PortalData/Responses/Ksan/ResponseStorageClass.cs +++ b/portal/Data/PortalData/Responses/Ksan/ResponseStorageClass.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Logs/ResponseSystemLog.cs b/portal/Data/PortalData/Responses/Logs/ResponseSystemLog.cs index 0887eafc..7011fa5d 100644 --- a/portal/Data/PortalData/Responses/Logs/ResponseSystemLog.cs +++ b/portal/Data/PortalData/Responses/Logs/ResponseSystemLog.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Logs/ResponseUserActionLog.cs b/portal/Data/PortalData/Responses/Logs/ResponseUserActionLog.cs index 9ec2459a..73fdda96 100644 --- a/portal/Data/PortalData/Responses/Logs/ResponseUserActionLog.cs +++ b/portal/Data/PortalData/Responses/Logs/ResponseUserActionLog.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterface.cs b/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterface.cs index b267a2c7..d66d0ea2 100644 --- a/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterface.cs +++ b/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterface.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterfaceDetail.cs b/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterfaceDetail.cs index f9409b04..75cd9b81 100644 --- a/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterfaceDetail.cs +++ b/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterfaceDetail.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterfaceVlan.cs b/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterfaceVlan.cs index ce6e3b1b..48c134f7 100644 --- a/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterfaceVlan.cs +++ b/portal/Data/PortalData/Responses/Networks/ResponseNetworkInterfaceVlan.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Region/ResponseRegion.cs b/portal/Data/PortalData/Responses/Region/ResponseRegion.cs index 0895b34f..8e8b6832 100644 --- a/portal/Data/PortalData/Responses/Region/ResponseRegion.cs +++ b/portal/Data/PortalData/Responses/Region/ResponseRegion.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Servers/ResponseServer.cs b/portal/Data/PortalData/Responses/Servers/ResponseServer.cs index 61cb8a2f..fbbc7b8b 100644 --- a/portal/Data/PortalData/Responses/Servers/ResponseServer.cs +++ b/portal/Data/PortalData/Responses/Servers/ResponseServer.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -54,7 +54,7 @@ public class ResponseServer : IModifier public decimal MemoryUsed { get; set; } /// 남은 메모리 크기 - public decimal MemoryFree { get; set; } + public decimal MemoryFree { get => MemoryTotal - MemoryUsed < 0 ? 0 : MemoryTotal - MemoryUsed; } /// 수정일시 public DateTime? ModDate { get; set; } = null; diff --git a/portal/Data/PortalData/Responses/Servers/ResponseServerDetail.cs b/portal/Data/PortalData/Responses/Servers/ResponseServerDetail.cs index 05716fe0..bdb49853 100644 --- a/portal/Data/PortalData/Responses/Servers/ResponseServerDetail.cs +++ b/portal/Data/PortalData/Responses/Servers/ResponseServerDetail.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Servers/ResponseServerWithNewWork.cs b/portal/Data/PortalData/Responses/Servers/ResponseServerWithNewWork.cs index df25b99d..d20451bd 100644 --- a/portal/Data/PortalData/Responses/Servers/ResponseServerWithNewWork.cs +++ b/portal/Data/PortalData/Responses/Servers/ResponseServerWithNewWork.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Services/ResponseSerivceMq.cs b/portal/Data/PortalData/Responses/Services/ResponseSerivceMq.cs index 9b32e499..3bcf0fbf 100644 --- a/portal/Data/PortalData/Responses/Services/ResponseSerivceMq.cs +++ b/portal/Data/PortalData/Responses/Services/ResponseSerivceMq.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Services/ResponseService.cs b/portal/Data/PortalData/Responses/Services/ResponseService.cs index 78ddf218..96318033 100644 --- a/portal/Data/PortalData/Responses/Services/ResponseService.cs +++ b/portal/Data/PortalData/Responses/Services/ResponseService.cs @@ -2,13 +2,14 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using PortalData.Enums; +using PortalData.Responses.Servers; namespace PortalData.Responses.Services { @@ -47,5 +48,8 @@ public class ResponseService : ResponseDataWithModifiedBy /// 스레드 수 public int ThreadCount { get; set; } + + /// 서버 정보 + public ResponseServer Server { get; set; } = new ResponseServer(); } } \ No newline at end of file diff --git a/portal/Data/PortalData/Responses/Services/ResponseServiceEvent.cs b/portal/Data/PortalData/Responses/Services/ResponseServiceEvent.cs index 88fe634b..f89fb007 100644 --- a/portal/Data/PortalData/Responses/Services/ResponseServiceEvent.cs +++ b/portal/Data/PortalData/Responses/Services/ResponseServiceEvent.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Services/ResponseServiceGroup.cs b/portal/Data/PortalData/Responses/Services/ResponseServiceGroup.cs index 5e5600d3..8204b124 100644 --- a/portal/Data/PortalData/Responses/Services/ResponseServiceGroup.cs +++ b/portal/Data/PortalData/Responses/Services/ResponseServiceGroup.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Services/ResponseServiceGroupWithServices.cs b/portal/Data/PortalData/Responses/Services/ResponseServiceGroupWithServices.cs index 4fecfecd..3ed2ef92 100644 --- a/portal/Data/PortalData/Responses/Services/ResponseServiceGroupWithServices.cs +++ b/portal/Data/PortalData/Responses/Services/ResponseServiceGroupWithServices.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Services/ResponseServiceWithGroup.cs b/portal/Data/PortalData/Responses/Services/ResponseServiceWithGroup.cs index 669bb303..98b7561e 100644 --- a/portal/Data/PortalData/Responses/Services/ResponseServiceWithGroup.cs +++ b/portal/Data/PortalData/Responses/Services/ResponseServiceWithGroup.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/Responses/Services/ResponseServiceWithVlans.cs b/portal/Data/PortalData/Responses/Services/ResponseServiceWithVlans.cs index 45092bf9..fd1404d1 100644 --- a/portal/Data/PortalData/Responses/Services/ResponseServiceWithVlans.cs +++ b/portal/Data/PortalData/Responses/Services/ResponseServiceWithVlans.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/UserExtension.cs b/portal/Data/PortalData/UserExtension.cs index a11ba051..1bf37d28 100644 --- a/portal/Data/PortalData/UserExtension.cs +++ b/portal/Data/PortalData/UserExtension.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalData/ValidationAttributes/IpAddressAttribute.cs b/portal/Data/PortalData/ValidationAttributes/IpAddressAttribute.cs index 331a7e87..7c69a545 100644 --- a/portal/Data/PortalData/ValidationAttributes/IpAddressAttribute.cs +++ b/portal/Data/PortalData/ValidationAttributes/IpAddressAttribute.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Data/PortalResources/Resource.Designer.cs b/portal/Data/PortalResources/Resource.Designer.cs index e5c18481..4f63acab 100644 --- a/portal/Data/PortalResources/Resource.Designer.cs +++ b/portal/Data/PortalResources/Resource.Designer.cs @@ -1375,11 +1375,11 @@ public static string EM_DISKS_REMOVE_AFTER_REMOVING_DISKPOOL { } /// - /// There is a service that uses that disk. Remove the disk from the service and delete it.과(와) 유사한 지역화된 문자열을 찾습니다. + /// There is a server that uses that disk. Remove the disk from the server and delete it.과(와) 유사한 지역화된 문자열을 찾습니다. /// - public static string EM_DISKS_REMOVE_AFTER_REMOVING_SERVICE { + public static string EM_DISKS_REMOVE_AFTER_REMOVING_SERVER { get { - return ResourceManager.GetString("EM_DISKS_REMOVE_AFTER_REMOVING_SERVICE", resourceCulture); + return ResourceManager.GetString("EM_DISKS_REMOVE_AFTER_REMOVING_SERVER", resourceCulture); } } @@ -1635,6 +1635,15 @@ public static string EM_SERVERS_REMOVE_AFTER_REMOVING_DISK { } } + /// + /// There is a service attached to that server. Delete the server after deleting the attached service.과(와) 유사한 지역화된 문자열을 찾습니다. + /// + public static string EM_SERVERS_REMOVE_AFTER_REMOVING_SERVICE { + get { + return ResourceManager.GetString("EM_SERVERS_REMOVE_AFTER_REMOVING_SERVICE", resourceCulture); + } + } + /// /// Please enter the server id.과(와) 유사한 지역화된 문자열을 찾습니다. /// @@ -3921,6 +3930,15 @@ public static string UL_DISKPOOL_REPLICA_TYPE { } } + /// + /// EC과(와) 유사한 지역화된 문자열을 찾습니다. + /// + public static string UL_DISKPOOL_REPLICA_TYPE_EC { + get { + return ResourceManager.GetString("UL_DISKPOOL_REPLICA_TYPE_EC", resourceCulture); + } + } + /// /// 1+1과(와) 유사한 지역화된 문자열을 찾습니다. /// diff --git a/portal/Data/PortalResources/Resource.ko.resx b/portal/Data/PortalResources/Resource.ko.resx index b8638df5..1dda3ba0 100644 --- a/portal/Data/PortalResources/Resource.ko.resx +++ b/portal/Data/PortalResources/Resource.ko.resx @@ -1141,11 +1141,14 @@ 해당 서버에 연결된 디스크가 존재합니다. 연결된 디스크를 삭제 후에 서버를 삭제하여 주세요. + + 해당 서버에 연결된 서비스가 존재합니다. 연결된 서비스를 삭제 후에 서버를 삭제하여 주세요. + 해당 디스크가 포함된 디스크 풀이 존재합니다. 해당 디스크 풀에서 디스크를 제거한 후 삭제하여 주세요. - - 해당 디스크를 사용하는 서비스가 존재합니다. 해당 서비스에서 디스크를 제거한 후 삭제하여 주세요. + + 해당 디스크를 사용하는 서버가 존재합니다. 해당 서버에서 디스크를 제거한 후 삭제하여 주세요. 스레드 수는 0 이상이어야 합니다. diff --git a/portal/Data/PortalResources/Resource.resx b/portal/Data/PortalResources/Resource.resx index 6d4ca65c..e90d58e1 100644 --- a/portal/Data/PortalResources/Resource.resx +++ b/portal/Data/PortalResources/Resource.resx @@ -1441,13 +1441,17 @@ There is a disk attached to that server. Delete the server after deleting the attached disk. 해당 서버에 연결된 디스크가 존재합니다. 연결된 디스크를 삭제 후에 서버를 삭제하여 주세요. + + There is a service attached to that server. Delete the server after deleting the attached service. + 해당 서버에 연결된 서비스가 존재합니다. 연결된 서비스를 삭제 후에 서버를 삭제하여 주세요. + A disk pool containing that disk exists. Remove the disk from the disk pool and delete it. 해당 디스크가 포함된 디스크 풀이 존재합니다. 해당 디스크 풀에서 디스크를 제거한 후 삭제하여 주세요. - - There is a service that uses that disk. Remove the disk from the service and delete it. - 해당 디스크를 사용하는 서비스가 존재합니다. 해당 서비스에서 디스크를 제거한 후 삭제하여 주세요. + + There is a server that uses that disk. Remove the disk from the server and delete it. + 해당 디스크를 사용하는 서버가 존재합니다. 해당 서버에서 디스크를 제거한 후 삭제하여 주세요. Thread count must be at least 0. @@ -1913,4 +1917,8 @@ DEFAULT_USER_SECRETKEY + + EC + ErasureCode + \ No newline at end of file diff --git a/portal/Models/PortalModels/PortalModels.AllowedConnectionIp.cs b/portal/Models/PortalModels/PortalModels.AllowedConnectionIp.cs index fbbd6af4..bd304afe 100644 --- a/portal/Models/PortalModels/PortalModels.AllowedConnectionIp.cs +++ b/portal/Models/PortalModels/PortalModels.AllowedConnectionIp.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.ApiKey.cs b/portal/Models/PortalModels/PortalModels.ApiKey.cs index e42c6d53..2dd6e4df 100644 --- a/portal/Models/PortalModels/PortalModels.ApiKey.cs +++ b/portal/Models/PortalModels/PortalModels.ApiKey.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.ClaimName.cs b/portal/Models/PortalModels/PortalModels.ClaimName.cs index 465b42e9..496b9a56 100644 --- a/portal/Models/PortalModels/PortalModels.ClaimName.cs +++ b/portal/Models/PortalModels/PortalModels.ClaimName.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.Config.cs b/portal/Models/PortalModels/PortalModels.Config.cs index 1b4473cc..b5672c2e 100644 --- a/portal/Models/PortalModels/PortalModels.Config.cs +++ b/portal/Models/PortalModels/PortalModels.Config.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.Disk.cs b/portal/Models/PortalModels/PortalModels.Disk.cs index c212c849..5abd49f9 100644 --- a/portal/Models/PortalModels/PortalModels.Disk.cs +++ b/portal/Models/PortalModels/PortalModels.Disk.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { @@ -28,7 +33,6 @@ public Disk() this.UsedSize = 0m; this.Read = 0m; this.Write = 0m; - this.ServiceDisks = new List(); this.DiskUsages = new List(); OnCreated(); } @@ -79,9 +83,6 @@ public Disk() /// 서버 정보 public virtual Server Server { get; set; } - /// 서비스 디스크 목록 - public virtual IList ServiceDisks { get; set; } - /// 디스크 풀 정보 public virtual DiskPool DiskPool { get; set; } diff --git a/portal/Models/PortalModels/PortalModels.DiskPool.cs b/portal/Models/PortalModels/PortalModels.DiskPool.cs index 94155098..27ca3701 100644 --- a/portal/Models/PortalModels/PortalModels.DiskPool.cs +++ b/portal/Models/PortalModels/PortalModels.DiskPool.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { @@ -67,6 +72,8 @@ public DiskPool() public virtual IList UserDiskPools { get; set; } + public virtual DiskPoolEC EC { get; set; } + #region Extensibility Method Definitions partial void OnCreated(); diff --git a/portal/Models/PortalModels/PortalModels.DiskUsage.cs b/portal/Models/PortalModels/PortalModels.DiskUsage.cs index ebe701b3..877ddb24 100644 --- a/portal/Models/PortalModels/PortalModels.DiskUsage.cs +++ b/portal/Models/PortalModels/PortalModels.DiskUsage.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.DiskpoolEC.cs b/portal/Models/PortalModels/PortalModels.DiskpoolEC.cs new file mode 100644 index 00000000..bed10c23 --- /dev/null +++ b/portal/Models/PortalModels/PortalModels.DiskpoolEC.cs @@ -0,0 +1,47 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License.See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; + +namespace PortalModels +{ + /// 디스크풀 EC 설정 + public partial class DiskPoolEC + { + + public DiskPoolEC() + { + OnCreated(); + } + + public virtual Guid DiskPoolId { get; set; } + + /// 원본 파일 등분 갯수 + public virtual int M { get; set; } + + /// 인코딩 갯수 + public virtual int K { get; set; } + + public virtual DiskPool DiskPool { get; set; } + + #region Extensibility Method Definitions + + partial void OnCreated(); + + #endregion + } + +} diff --git a/portal/Models/PortalModels/PortalModels.EnumDbDiskPoolReplicaType.cs b/portal/Models/PortalModels/PortalModels.EnumDbDiskPoolReplicaType.cs index 3fbb5fcb..e3d2811a 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbDiskPoolReplicaType.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbDiskPoolReplicaType.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { @@ -16,6 +23,7 @@ public enum EnumDbDiskPoolReplicaType : int { OnePlusZero = 1, OnePlusOne, - OnePlusTwo + OnePlusTwo, + ErasureCode } } diff --git a/portal/Models/PortalModels/PortalModels.EnumDbDiskPoolType.cs b/portal/Models/PortalModels/PortalModels.EnumDbDiskPoolType.cs index fff78c1c..eb879e0a 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbDiskPoolType.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbDiskPoolType.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbDiskRwMode.cs b/portal/Models/PortalModels/PortalModels.EnumDbDiskRwMode.cs index 6141e91c..34ad2692 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbDiskRwMode.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbDiskRwMode.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbDiskState.cs b/portal/Models/PortalModels/PortalModels.EnumDbDiskState.cs index 3317a122..338c72ec 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbDiskState.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbDiskState.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbHaAction.cs b/portal/Models/PortalModels/PortalModels.EnumDbHaAction.cs index 6f92d237..c1a84a1b 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbHaAction.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbHaAction.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbLogLevel.cs b/portal/Models/PortalModels/PortalModels.EnumDbLogLevel.cs index e870dc84..2bdeddbc 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbLogLevel.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbLogLevel.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbNetworkLinkState.cs b/portal/Models/PortalModels/PortalModels.EnumDbNetworkLinkState.cs index 5fbfd936..de6f5819 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbNetworkLinkState.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbNetworkLinkState.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbOnOff.cs b/portal/Models/PortalModels/PortalModels.EnumDbOnOff.cs index e406eb59..bf78e3f3 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbOnOff.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbOnOff.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbServerState.cs b/portal/Models/PortalModels/PortalModels.EnumDbServerState.cs index 693dd0e8..41e8e43c 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbServerState.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbServerState.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbServiceEventType.cs b/portal/Models/PortalModels/PortalModels.EnumDbServiceEventType.cs index 26d33cbf..d656a014 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbServiceEventType.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbServiceEventType.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbServiceGroupMode.cs b/portal/Models/PortalModels/PortalModels.EnumDbServiceGroupMode.cs index 1e698e96..481e6cbc 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbServiceGroupMode.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbServiceGroupMode.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbServiceState.cs b/portal/Models/PortalModels/PortalModels.EnumDbServiceState.cs index 3e5e407a..a93aeb5b 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbServiceState.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbServiceState.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.EnumDbServiceType.cs b/portal/Models/PortalModels/PortalModels.EnumDbServiceType.cs index 3dbd0687..5def6e55 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbServiceType.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbServiceType.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { @@ -26,9 +33,8 @@ public enum EnumDbServiceType : int ksanOSD, ksanGW, ksanRecovery, - ksanLifecycle, - ksanReplication, - ksanLogManager, - ksanMetering + ksanLifecycleManager, + ksanReplicationManager, + ksanLogManager } } diff --git a/portal/Models/PortalModels/PortalModels.EnumDbYesNo.cs b/portal/Models/PortalModels/PortalModels.EnumDbYesNo.cs index fb4fa074..19f25798 100644 --- a/portal/Models/PortalModels/PortalModels.EnumDbYesNo.cs +++ b/portal/Models/PortalModels/PortalModels.EnumDbYesNo.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.KsanUser.cs b/portal/Models/PortalModels/PortalModels.KsanUser.cs index 687cf47c..ef4dfe1b 100644 --- a/portal/Models/PortalModels/PortalModels.KsanUser.cs +++ b/portal/Models/PortalModels/PortalModels.KsanUser.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.NetworkInterface.cs b/portal/Models/PortalModels/PortalModels.NetworkInterface.cs index 6dab9638..75318152 100644 --- a/portal/Models/PortalModels/PortalModels.NetworkInterface.cs +++ b/portal/Models/PortalModels/PortalModels.NetworkInterface.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.NetworkInterfaceUsage.cs b/portal/Models/PortalModels/PortalModels.NetworkInterfaceUsage.cs index 5fb3a2e2..7160624b 100644 --- a/portal/Models/PortalModels/PortalModels.NetworkInterfaceUsage.cs +++ b/portal/Models/PortalModels/PortalModels.NetworkInterfaceUsage.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.NetworkInterfaceVlan.cs b/portal/Models/PortalModels/PortalModels.NetworkInterfaceVlan.cs index a415852f..be0796c1 100644 --- a/portal/Models/PortalModels/PortalModels.NetworkInterfaceVlan.cs +++ b/portal/Models/PortalModels/PortalModels.NetworkInterfaceVlan.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.PortalModel.Extension.cs b/portal/Models/PortalModels/PortalModels.PortalModel.Extension.cs index f2d7f871..bdf6ffc8 100644 --- a/portal/Models/PortalModels/PortalModels.PortalModel.Extension.cs +++ b/portal/Models/PortalModels/PortalModels.PortalModel.Extension.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. diff --git a/portal/Models/PortalModels/PortalModels.PortalModel.cs b/portal/Models/PortalModels/PortalModels.PortalModel.cs index 1f5ec257..eeec92fb 100644 --- a/portal/Models/PortalModels/PortalModels.PortalModel.cs +++ b/portal/Models/PortalModels/PortalModels.PortalModel.cs @@ -2,22 +2,31 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; using System.Linq; +using System.Linq.Expressions; +using System.Reflection; +using System.Threading.Tasks; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Internal; +using Microsoft.EntityFrameworkCore.Metadata; namespace PortalModels { public partial class PortalModel : DbContext { - public PortalModel() : base() { OnCreated(); @@ -33,7 +42,7 @@ protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) { if (!optionsBuilder.IsConfigured || (!optionsBuilder.Options.Extensions.OfType().Any(ext => !string.IsNullOrEmpty(ext.ConnectionString) || ext.Connection != null) && - !optionsBuilder.Options.Extensions.Any(ext => !(ext is RelationalOptionsExtension) && !(ext is CoreOptionsExtension)))) + !optionsBuilder.Options.Extensions.Any(ext => !(ext is RelationalOptionsExtension) && !(ext is CoreOptionsExtension)))) { } CustomizeConfiguration(ref optionsBuilder); @@ -80,8 +89,6 @@ protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) public virtual DbSet Disks { get; set; } - public virtual DbSet ServiceDisks { get; set; } - public virtual DbSet DiskPools { get; set; } public virtual DbSet ServerUsages { get; set; } @@ -102,6 +109,10 @@ protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) public virtual DbSet ServiceEventLogs { get; set; } + public virtual DbSet S3Loggings { get; set; } + + public virtual DbSet DiskPoolECs { get; set; } + protected override void OnModelCreating(ModelBuilder modelBuilder) { base.OnModelCreating(modelBuilder); @@ -163,9 +174,6 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) this.DiskMapping(modelBuilder); this.CustomizeDiskMapping(modelBuilder); - this.ServiceDiskMapping(modelBuilder); - this.CustomizeServiceDiskMapping(modelBuilder); - this.DiskPoolMapping(modelBuilder); this.CustomizeDiskPoolMapping(modelBuilder); @@ -196,6 +204,12 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) this.ServiceEventLogMapping(modelBuilder); this.CustomizeServiceEventLogMapping(modelBuilder); + this.S3LoggingMapping(modelBuilder); + this.CustomizeS3LoggingMapping(modelBuilder); + + this.DiskPoolECMapping(modelBuilder); + this.CustomizeDiskPoolECMapping(modelBuilder); + RelationshipsMapping(modelBuilder); CustomizeMapping(ref modelBuilder); } @@ -601,20 +615,6 @@ private void DiskMapping(ModelBuilder modelBuilder) #endregion - #region ServiceDisk Mapping - - private void ServiceDiskMapping(ModelBuilder modelBuilder) - { - modelBuilder.Entity().ToTable(@"SERVICE_DISKS"); - modelBuilder.Entity().Property(x => x.ServiceId).HasColumnName(@"SERVICE_ID").IsRequired().ValueGeneratedNever(); - modelBuilder.Entity().Property(x => x.DiskId).HasColumnName(@"DISK_ID").IsRequired().ValueGeneratedNever(); - modelBuilder.Entity().HasKey(@"ServiceId", @"DiskId"); - } - - partial void CustomizeServiceDiskMapping(ModelBuilder modelBuilder); - - #endregion - #region DiskPool Mapping private void DiskPoolMapping(ModelBuilder modelBuilder) @@ -797,6 +797,58 @@ private void ServiceEventLogMapping(ModelBuilder modelBuilder) #endregion + #region S3Logging Mapping + + private void S3LoggingMapping(ModelBuilder modelBuilder) + { + modelBuilder.Entity().ToTable(@"S3LOGGING"); + modelBuilder.Entity().Property(x => x.Id).HasColumnName(@"ID").IsRequired().ValueGeneratedOnAdd(); + modelBuilder.Entity().Property(x => x.UserName).HasColumnName(@"USER_NAME").HasColumnType(@"varchar(64)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.BucketName).HasColumnName(@"BUCKET_NAME").HasColumnType(@"varchar(64)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.InDate).HasColumnName(@"IN_DATE").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.RemoteHost).HasColumnName(@"REMOTE_HOST").HasColumnType(@"varchar(256)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.RequestUser).HasColumnName(@"REQUEST_USER").HasColumnType(@"varchar(64)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.RequestId).HasColumnName(@"REQUEST_ID").HasColumnType(@"varchar(64)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.Operation).HasColumnName(@"OPERATION").HasColumnType(@"varchar(64)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.ObjectName).HasColumnName(@"OBJECT_NAME").HasColumnType(@"varchar(2048)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.RequestUri).HasColumnName(@"REQUEST_URI").HasColumnType(@"varchar(2048)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.StatusCode).HasColumnName(@"STATUS_CODE").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.ErrorCode).HasColumnName(@"ERROR_CODE").HasColumnType(@"varchar(256)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.ResponseLength).HasColumnName(@"RESPONSE_LENGTH").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.ObjectLength).HasColumnName(@"OBJECT_LENGTH").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.TotalTime).HasColumnName(@"TOTAL_TIME").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.RequestLength).HasColumnName(@"REQUEST_LENGTH").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.Rererer).HasColumnName(@"RERERER").HasColumnType(@"varchar(64)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.UserAgent).HasColumnName(@"USER_AGENT").HasColumnType(@"varchar(256)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.VersionId).HasColumnName(@"VERSION_ID").HasColumnType(@"varchar(64)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.HostId).HasColumnName(@"HOST_ID").HasColumnType(@"varchar(256)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.Sign).HasColumnName(@"SIGN").HasColumnType(@"varchar(32)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.SslGroup).HasColumnName(@"SSL_GROUP").HasColumnType(@"varchar(64)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.SignType).HasColumnName(@"SIGN_TYPE").HasColumnType(@"varchar(32)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.Endpoint).HasColumnName(@"ENDPOINT").HasColumnType(@"varchar(64)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.TlsVersion).HasColumnName(@"TLS_VERSION").HasColumnType(@"varchar(32)").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().HasKey(@"Id"); + } + + partial void CustomizeS3LoggingMapping(ModelBuilder modelBuilder); + + #endregion + + #region DiskPoolEC Mapping + + private void DiskPoolECMapping(ModelBuilder modelBuilder) + { + modelBuilder.Entity().ToTable(@"DISK_POOL_ECS"); + modelBuilder.Entity().Property(x => x.DiskPoolId).HasColumnName(@"DISK_POOL_ID").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.M).HasColumnName(@"M").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().Property(x => x.K).HasColumnName(@"K").IsRequired().ValueGeneratedNever(); + modelBuilder.Entity().HasKey(@"DiskPoolId"); + } + + partial void CustomizeDiskPoolECMapping(ModelBuilder modelBuilder); + + #endregion + private void RelationshipsMapping(ModelBuilder modelBuilder) { modelBuilder.Entity().HasOne(x => x.Server).WithMany(op => op.NetworkInterfaces).HasForeignKey(@"ServerId").IsRequired(true); @@ -814,9 +866,8 @@ private void RelationshipsMapping(ModelBuilder modelBuilder) modelBuilder.Entity().HasOne(x => x.ServiceGroup).WithMany(op => op.Services).HasForeignKey(@"GroupId").IsRequired(false); modelBuilder.Entity().HasOne(x => x.RegUser).WithMany(op => op.RegServices).HasForeignKey(@"RegId").IsRequired(false); modelBuilder.Entity().HasOne(x => x.ModUser).WithMany(op => op.ModServices).HasForeignKey(@"ModId").IsRequired(false); - modelBuilder.Entity().HasMany(x => x.ServiceDisks).WithOne(op => op.Service).HasForeignKey(@"ServiceId").IsRequired(true); modelBuilder.Entity().HasMany(x => x.ServiceUsages).WithOne(op => op.Service).OnDelete(DeleteBehavior.Cascade).HasForeignKey(@"Id").IsRequired(true); - modelBuilder.Entity().HasOne(x => x.Server).WithMany(op => op.Services).HasForeignKey(@"ServerId").IsRequired(true); + modelBuilder.Entity().HasOne(x => x.Server).WithMany(op => op.Services).OnDelete(DeleteBehavior.Restrict).HasForeignKey(@"ServerId").IsRequired(true); modelBuilder.Entity().HasMany(x => x.ServiceEventLogs).WithOne(op => op.Service).HasForeignKey(@"Id").IsRequired(true); modelBuilder.Entity().HasOne(x => x.Service).WithMany(op => op.Vlans).HasForeignKey(@"ServiceId").IsRequired(true); @@ -824,9 +875,9 @@ private void RelationshipsMapping(ModelBuilder modelBuilder) modelBuilder.Entity().HasMany(x => x.NetworkInterfaces).WithOne(op => op.Server).HasForeignKey(@"ServerId").IsRequired(true); modelBuilder.Entity().HasOne(x => x.ModUser).WithMany(op => op.ModServers).HasForeignKey(@"ModId").IsRequired(false); - modelBuilder.Entity().HasMany(x => x.Disks).WithOne(op => op.Server).HasForeignKey(@"ServerId").IsRequired(true); + modelBuilder.Entity().HasMany(x => x.Disks).WithOne(op => op.Server).OnDelete(DeleteBehavior.Restrict).HasForeignKey(@"ServerId").IsRequired(true); modelBuilder.Entity().HasMany(x => x.ServerUsages).WithOne(op => op.Server).OnDelete(DeleteBehavior.Cascade).HasForeignKey(@"Id").IsRequired(true); - modelBuilder.Entity().HasMany(x => x.Services).WithOne(op => op.Server).HasForeignKey(@"ServerId").IsRequired(true); + modelBuilder.Entity().HasMany(x => x.Services).WithOne(op => op.Server).OnDelete(DeleteBehavior.Restrict).HasForeignKey(@"ServerId").IsRequired(true); modelBuilder.Entity().HasMany(x => x.Services).WithOne(op => op.ServiceGroup).HasForeignKey(@"GroupId").IsRequired(false); modelBuilder.Entity().HasOne(x => x.RegUser).WithMany(op => op.RegServiceGroups).HasForeignKey(@"RegId").IsRequired(false); @@ -865,16 +916,13 @@ private void RelationshipsMapping(ModelBuilder modelBuilder) modelBuilder.Entity().HasOne(x => x.User).WithMany(op => op.UserActionLogs).OnDelete(DeleteBehavior.Cascade).HasForeignKey(@"UserId").IsRequired(true); - modelBuilder.Entity().HasOne(x => x.Server).WithMany(op => op.Disks).HasForeignKey(@"ServerId").IsRequired(true); - modelBuilder.Entity().HasMany(x => x.ServiceDisks).WithOne(op => op.Disk).OnDelete(DeleteBehavior.Cascade).HasForeignKey(@"DiskId").IsRequired(true); + modelBuilder.Entity().HasOne(x => x.Server).WithMany(op => op.Disks).OnDelete(DeleteBehavior.Restrict).HasForeignKey(@"ServerId").IsRequired(true); modelBuilder.Entity().HasOne(x => x.DiskPool).WithMany(op => op.Disks).HasForeignKey(@"DiskPoolId").IsRequired(false); modelBuilder.Entity().HasMany(x => x.DiskUsages).WithOne(op => op.Disk).OnDelete(DeleteBehavior.Cascade).HasForeignKey(@"Id").IsRequired(true); - modelBuilder.Entity().HasOne(x => x.Service).WithMany(op => op.ServiceDisks).HasForeignKey(@"ServiceId").IsRequired(true); - modelBuilder.Entity().HasOne(x => x.Disk).WithMany(op => op.ServiceDisks).OnDelete(DeleteBehavior.Cascade).HasForeignKey(@"DiskId").IsRequired(true); - modelBuilder.Entity().HasMany(x => x.Disks).WithOne(op => op.DiskPool).HasForeignKey(@"DiskPoolId").IsRequired(false); modelBuilder.Entity().HasMany(x => x.UserDiskPools).WithOne(op => op.DiskPool).HasForeignKey(@"DiskPoolId").IsRequired(true); + modelBuilder.Entity().HasOne(x => x.EC).WithOne(op => op.DiskPool).OnDelete(DeleteBehavior.Cascade).HasForeignKey(typeof(DiskPoolEC), @"DiskPoolId").IsRequired(false); modelBuilder.Entity().HasOne(x => x.Server).WithMany(op => op.ServerUsages).OnDelete(DeleteBehavior.Cascade).HasForeignKey(@"Id").IsRequired(true); @@ -890,6 +938,8 @@ private void RelationshipsMapping(ModelBuilder modelBuilder) modelBuilder.Entity().HasOne(x => x.Disk).WithMany(op => op.DiskUsages).OnDelete(DeleteBehavior.Cascade).HasForeignKey(@"Id").IsRequired(true); modelBuilder.Entity().HasOne(x => x.Service).WithMany(op => op.ServiceEventLogs).HasForeignKey(@"Id").IsRequired(true); + + modelBuilder.Entity().HasOne(x => x.DiskPool).WithOne(op => op.EC).OnDelete(DeleteBehavior.Cascade).HasForeignKey(typeof(DiskPoolEC), @"DiskPoolId").IsRequired(true); } partial void CustomizeMapping(ref ModelBuilder modelBuilder); diff --git a/portal/Models/PortalModels/PortalModels.PortalModelDiagram.view b/portal/Models/PortalModels/PortalModels.PortalModelDiagram.view index e71e0f9c..3cf8c05f 100644 --- a/portal/Models/PortalModels/PortalModels.PortalModelDiagram.view +++ b/portal/Models/PortalModels/PortalModels.PortalModelDiagram.view @@ -1,4 +1,4 @@ - + @@ -563,8 +563,8 @@ - 1869 px - 335 px + 1440 px + 356 px 150 px @@ -930,8 +930,8 @@ - 1869 px - 948 px + 1440 px + 969 px 150 px @@ -1409,7 +1409,7 @@ 18 px - f459e10a-8674-4147-b301-735c7dfaa3a0 + 1496669b-ed1d-40cc-867a-091f0bfa8eee EntityDeveloper.EntityFrameworkCore.EntityRelationProperty @@ -1428,7 +1428,7 @@ 18 px - 1496669b-ed1d-40cc-867a-091f0bfa8eee + 7a1104d6-325d-4cd9-893b-e4a1fbe500e0 EntityDeveloper.EntityFrameworkCore.EntityRelationProperty @@ -1446,25 +1446,6 @@ 144 px 18 px - - 7a1104d6-325d-4cd9-893b-e4a1fbe500e0 - EntityDeveloper.EntityFrameworkCore.EntityRelationProperty - - - - - 78 - 70 - - - - 0 px - 126 px - - - 144 px - 18 px - 4dcc1299-0080-4749-9b69-5f1df8b62967 EntityDeveloper.EntityFrameworkCore.EntityRelationProperty @@ -1477,30 +1458,30 @@ 145 px - 163 px + 145 px 0 px - 163 px + 145 px false - 1864 px - 1560 px + 1435 px + 1581 px 150 px - 536 px + 518 px 100 px - 536 px + 518 px 800 px - 536 px + 518 px @@ -1511,20 +1492,20 @@ - 79 + 78 0 - 80 - 79 + 79 + 78 - 81 - 80 + 80 + 79 @@ -1542,8 +1523,8 @@ - 82 - 80 + 81 + 79 @@ -1571,14 +1552,14 @@ - 83 - 79 + 82 + 78 - 84 - 83 + 83 + 82 @@ -1592,8 +1573,8 @@ - 85 - 83 + 84 + 82 @@ -1626,8 +1607,8 @@ - 1869 px - 1363 px + 1440 px + 1384 px 150 px @@ -1650,20 +1631,20 @@ - 86 + 85 0 - 87 - 86 + 86 + 85 - 88 - 87 + 87 + 86 @@ -1681,8 +1662,8 @@ - 89 - 87 + 88 + 86 @@ -1700,8 +1681,8 @@ - 90 - 87 + 89 + 86 @@ -1719,8 +1700,8 @@ - 91 - 87 + 90 + 86 @@ -1738,8 +1719,8 @@ - 92 - 87 + 91 + 86 @@ -1757,8 +1738,8 @@ - 93 - 87 + 92 + 86 @@ -1776,8 +1757,8 @@ - 94 - 87 + 93 + 86 @@ -1795,8 +1776,8 @@ - 95 - 87 + 94 + 86 @@ -1814,8 +1795,8 @@ - 96 - 87 + 95 + 86 @@ -1833,8 +1814,8 @@ - 97 - 87 + 96 + 86 @@ -1852,8 +1833,8 @@ - 98 - 87 + 97 + 86 @@ -1871,8 +1852,8 @@ - 99 - 87 + 98 + 86 @@ -1890,8 +1871,8 @@ - 100 - 87 + 99 + 86 @@ -1909,8 +1890,8 @@ - 101 - 87 + 100 + 86 @@ -1928,8 +1909,8 @@ - 102 - 87 + 101 + 86 @@ -1957,14 +1938,14 @@ - 103 - 86 + 102 + 85 - 104 - 103 + 103 + 102 @@ -1978,8 +1959,8 @@ - 105 - 103 + 104 + 102 @@ -1997,8 +1978,8 @@ - 106 - 103 + 105 + 102 @@ -2016,8 +1997,8 @@ - 107 - 103 + 106 + 102 @@ -2035,8 +2016,8 @@ - 108 - 103 + 107 + 102 @@ -2069,8 +2050,8 @@ - 2288 px - 407 px + 1880 px + 280 px 150 px @@ -2093,20 +2074,20 @@ - 109 + 108 0 - 110 - 109 + 109 + 108 - 111 - 110 + 110 + 109 @@ -2124,8 +2105,8 @@ - 112 - 110 + 111 + 109 @@ -2143,8 +2124,8 @@ - 113 - 110 + 112 + 109 @@ -2162,8 +2143,8 @@ - 114 - 110 + 113 + 109 @@ -2181,8 +2162,8 @@ - 115 - 110 + 114 + 109 @@ -2200,8 +2181,8 @@ - 116 - 110 + 115 + 109 @@ -2219,8 +2200,8 @@ - 117 - 110 + 116 + 109 @@ -2238,8 +2219,8 @@ - 118 - 110 + 117 + 109 @@ -2257,8 +2238,8 @@ - 119 - 110 + 118 + 109 @@ -2276,8 +2257,8 @@ - 120 - 110 + 119 + 109 @@ -2295,8 +2276,8 @@ - 121 - 110 + 120 + 109 @@ -2324,14 +2305,14 @@ - 122 - 109 + 121 + 108 - 123 - 122 + 122 + 121 @@ -2345,8 +2326,8 @@ - 124 - 122 + 123 + 121 @@ -2364,8 +2345,8 @@ - 125 - 122 + 124 + 121 @@ -2398,8 +2379,8 @@ - 1481 px - 1368 px + 1120 px + 1360 px 150 px @@ -2422,20 +2403,20 @@ - 126 + 125 0 - 127 - 126 + 126 + 125 - 128 - 127 + 127 + 126 @@ -2453,8 +2434,8 @@ - 129 - 127 + 128 + 126 @@ -2472,8 +2453,8 @@ - 130 - 127 + 129 + 126 @@ -2491,8 +2472,8 @@ - 131 - 127 + 130 + 126 @@ -2510,8 +2491,8 @@ - 132 - 127 + 131 + 126 @@ -2539,8 +2520,8 @@ - 133 - 126 + 132 + 125 @@ -2559,8 +2540,8 @@ - 2504 px - 336 px + 40 px + 400 px 150 px @@ -2583,20 +2564,20 @@ - 134 + 133 0 - 135 - 134 + 134 + 133 - 136 - 135 + 135 + 134 @@ -2614,8 +2595,8 @@ - 137 - 135 + 136 + 134 @@ -2643,8 +2624,8 @@ - 138 - 134 + 137 + 133 @@ -2663,8 +2644,8 @@ - 3104 px - 288 px + 240 px + 400 px 150 px @@ -2687,20 +2668,20 @@ - 139 + 138 0 - 140 - 139 + 139 + 138 - 141 - 140 + 140 + 139 @@ -2718,8 +2699,8 @@ - 142 - 140 + 141 + 139 @@ -2737,8 +2718,8 @@ - 143 - 140 + 142 + 139 @@ -2756,8 +2737,8 @@ - 144 - 140 + 143 + 139 @@ -2785,8 +2766,8 @@ - 145 - 139 + 144 + 138 @@ -2805,8 +2786,8 @@ - 3104 px - 72 px + 440 px + 400 px 150 px @@ -2829,20 +2810,20 @@ - 146 + 145 0 - 147 - 146 + 146 + 145 - 148 - 147 + 147 + 146 @@ -2860,8 +2841,8 @@ - 149 - 147 + 148 + 146 @@ -2879,8 +2860,8 @@ - 150 - 147 + 149 + 146 @@ -2898,8 +2879,8 @@ - 151 - 147 + 150 + 146 @@ -2917,8 +2898,8 @@ - 152 - 147 + 151 + 146 @@ -2936,8 +2917,8 @@ - 153 - 147 + 152 + 146 @@ -2955,8 +2936,8 @@ - 154 - 147 + 153 + 146 @@ -2974,8 +2955,8 @@ - 155 - 147 + 154 + 146 @@ -3003,14 +2984,14 @@ - 156 - 146 + 155 + 145 - 157 - 156 + 156 + 155 @@ -3024,8 +3005,8 @@ - 158 - 156 + 157 + 155 @@ -3058,8 +3039,8 @@ - 677 px - 927 px + 643 px + 968 px 150 px @@ -3082,20 +3063,20 @@ - 159 + 158 0 - 160 - 159 + 159 + 158 - 161 - 160 + 160 + 159 @@ -3113,8 +3094,8 @@ - 162 - 160 + 161 + 159 @@ -3142,14 +3123,14 @@ - 163 - 159 + 162 + 158 - 164 - 163 + 163 + 162 @@ -3178,8 +3159,8 @@ - 309 px - 1904 px + 48 px + 1640 px 150 px @@ -3202,20 +3183,20 @@ - 165 + 164 0 - 166 - 165 + 165 + 164 - 167 - 166 + 166 + 165 @@ -3233,8 +3214,8 @@ - 168 - 166 + 167 + 165 @@ -3252,8 +3233,8 @@ - 169 - 166 + 168 + 165 @@ -3271,8 +3252,8 @@ - 170 - 166 + 169 + 165 @@ -3300,14 +3281,14 @@ - 171 - 165 + 170 + 164 - 172 - 171 + 171 + 170 @@ -3336,8 +3317,8 @@ - 40 px - 1435 px + 48 px + 1440 px 150 px @@ -3360,20 +3341,20 @@ - 173 + 172 0 - 174 - 173 + 173 + 172 - 175 - 174 + 174 + 173 @@ -3391,8 +3372,8 @@ - 176 - 174 + 175 + 173 @@ -3410,8 +3391,8 @@ - 177 - 174 + 176 + 173 @@ -3429,8 +3410,8 @@ - 178 - 174 + 177 + 173 @@ -3458,14 +3439,14 @@ - 179 - 173 + 178 + 172 - 180 - 179 + 179 + 178 @@ -3479,8 +3460,8 @@ - 181 - 179 + 180 + 178 @@ -3498,8 +3479,8 @@ - 182 - 179 + 181 + 178 @@ -3532,7 +3513,7 @@ - 677 px + 640 px 674 px @@ -3556,20 +3537,20 @@ - 183 + 182 0 - 184 - 183 + 183 + 182 - 185 - 184 + 184 + 183 @@ -3587,8 +3568,8 @@ - 186 - 184 + 185 + 183 @@ -3606,8 +3587,8 @@ - 187 - 184 + 186 + 183 @@ -3625,8 +3606,8 @@ - 188 - 184 + 187 + 183 @@ -3654,14 +3635,14 @@ - 189 - 183 + 188 + 182 - 190 - 189 + 189 + 188 @@ -3690,7 +3671,7 @@ - 677 px + 640 px 456 px @@ -3714,20 +3695,20 @@ - 191 + 190 0 - 192 - 191 + 191 + 190 - 193 - 192 + 192 + 191 @@ -3745,8 +3726,8 @@ - 194 - 192 + 193 + 191 @@ -3764,8 +3745,8 @@ - 195 - 192 + 194 + 191 @@ -3783,8 +3764,8 @@ - 196 - 192 + 195 + 191 @@ -3802,8 +3783,8 @@ - 197 - 192 + 196 + 191 @@ -3821,8 +3802,8 @@ - 198 - 192 + 197 + 191 @@ -3850,8 +3831,8 @@ - 199 - 191 + 198 + 190 @@ -3870,8 +3851,8 @@ - 2698 px - 39 px + 240 px + 200 px 150 px @@ -3894,20 +3875,20 @@ - 200 + 199 0 - 201 - 200 + 200 + 199 - 202 - 201 + 201 + 200 @@ -3925,8 +3906,8 @@ - 203 - 201 + 202 + 200 @@ -3954,14 +3935,14 @@ - 204 - 200 + 203 + 199 - 205 - 204 + 204 + 203 @@ -3975,8 +3956,8 @@ - 206 - 204 + 205 + 203 @@ -4009,8 +3990,8 @@ - 467 px - 701 px + 320 px + 720 px 150 px @@ -4033,20 +4014,20 @@ - 207 + 206 0 - 208 - 207 + 207 + 206 - 209 - 208 + 208 + 207 @@ -4064,8 +4045,8 @@ - 210 - 208 + 209 + 207 @@ -4083,8 +4064,8 @@ - 211 - 208 + 210 + 207 @@ -4102,8 +4083,8 @@ - 212 - 208 + 211 + 207 @@ -4121,8 +4102,8 @@ - 213 - 208 + 212 + 207 @@ -4140,8 +4121,8 @@ - 214 - 208 + 213 + 207 @@ -4159,8 +4140,8 @@ - 215 - 208 + 214 + 207 @@ -4178,8 +4159,8 @@ - 216 - 208 + 215 + 207 @@ -4197,8 +4178,8 @@ - 217 - 208 + 216 + 207 @@ -4216,8 +4197,8 @@ - 218 - 208 + 217 + 207 @@ -4235,8 +4216,8 @@ - 219 - 208 + 218 + 207 @@ -4254,8 +4235,8 @@ - 220 - 208 + 219 + 207 @@ -4273,8 +4254,8 @@ - 221 - 208 + 220 + 207 @@ -4292,8 +4273,8 @@ - 222 - 208 + 221 + 207 @@ -4311,8 +4292,8 @@ - 223 - 208 + 222 + 207 @@ -4330,8 +4311,8 @@ - 224 - 208 + 223 + 207 @@ -4349,8 +4330,8 @@ - 225 - 208 + 224 + 207 @@ -4368,8 +4349,8 @@ - 226 - 208 + 225 + 207 @@ -4387,8 +4368,8 @@ - 227 - 208 + 226 + 207 @@ -4416,14 +4397,14 @@ - 228 - 207 + 227 + 206 - 229 - 228 + 228 + 227 @@ -4437,8 +4418,8 @@ - 230 - 228 + 229 + 227 @@ -4456,8 +4437,8 @@ - 231 - 228 + 230 + 227 @@ -4475,8 +4456,8 @@ - 232 - 228 + 231 + 227 @@ -4494,8 +4475,8 @@ - 233 - 228 + 232 + 227 @@ -4513,8 +4494,8 @@ - 234 - 228 + 233 + 227 @@ -4532,8 +4513,8 @@ - 235 - 228 + 234 + 227 @@ -4551,8 +4532,8 @@ - 236 - 228 + 235 + 227 @@ -4570,8 +4551,8 @@ - 237 - 228 + 236 + 227 @@ -4589,8 +4570,8 @@ - 238 - 228 + 237 + 227 @@ -4608,8 +4589,8 @@ - 239 - 228 + 238 + 227 @@ -4627,8 +4608,8 @@ - 240 - 228 + 239 + 227 @@ -4646,8 +4627,8 @@ - 241 - 228 + 240 + 227 @@ -4665,8 +4646,8 @@ - 242 - 228 + 241 + 227 @@ -4723,20 +4704,20 @@ - 243 + 242 0 - 244 - 243 + 243 + 242 - 245 - 244 + 244 + 243 @@ -4754,8 +4735,8 @@ - 246 - 244 + 245 + 243 @@ -4773,8 +4754,8 @@ - 247 - 244 + 246 + 243 @@ -4792,8 +4773,8 @@ - 248 - 244 + 247 + 243 @@ -4811,8 +4792,8 @@ - 249 - 244 + 248 + 243 @@ -4830,8 +4811,8 @@ - 250 - 244 + 249 + 243 @@ -4849,8 +4830,8 @@ - 251 - 244 + 250 + 243 @@ -4878,14 +4859,14 @@ - 252 - 243 + 251 + 242 - 253 - 252 + 252 + 251 @@ -4914,8 +4895,8 @@ - 243 px - 499 px + 48 px + 1200 px 150 px @@ -4938,20 +4919,20 @@ - 254 + 253 0 - 255 - 254 + 254 + 253 - 256 - 255 + 255 + 254 @@ -4969,8 +4950,8 @@ - 257 - 255 + 256 + 254 @@ -4988,8 +4969,8 @@ - 258 - 255 + 257 + 254 @@ -5007,8 +4988,8 @@ - 259 - 255 + 258 + 254 @@ -5026,8 +5007,8 @@ - 260 - 255 + 259 + 254 @@ -5045,8 +5026,8 @@ - 261 - 255 + 260 + 254 @@ -5064,8 +5045,8 @@ - 262 - 255 + 261 + 254 @@ -5083,8 +5064,8 @@ - 263 - 255 + 262 + 254 @@ -5102,8 +5083,8 @@ - 264 - 255 + 263 + 254 @@ -5121,8 +5102,8 @@ - 265 - 255 + 264 + 254 @@ -5140,8 +5121,8 @@ - 266 - 255 + 265 + 254 @@ -5159,8 +5140,8 @@ - 267 - 255 + 266 + 254 @@ -5178,8 +5159,8 @@ - 268 - 255 + 267 + 254 @@ -5197,8 +5178,8 @@ - 269 - 255 + 268 + 254 @@ -5216,8 +5197,8 @@ - 270 - 255 + 269 + 254 @@ -5245,14 +5226,14 @@ - 271 - 254 + 270 + 253 - 272 - 271 + 271 + 270 @@ -5266,8 +5247,8 @@ - 273 - 271 + 272 + 270 @@ -5278,25 +5259,6 @@ 144 px 18 px - - 07dbefd3-48b5-4681-b806-53c9aa417fc9 - EntityDeveloper.EntityFrameworkCore.EntityRelationProperty - - - - - 274 - 271 - - - - 0 px - 36 px - - - 144 px - 18 px - 24adca67-209e-4a95-bdd7-a9e883cd76d9 EntityDeveloper.EntityFrameworkCore.EntityRelationProperty @@ -5304,13 +5266,13 @@ - 275 - 271 + 273 + 270 0 px - 54 px + 36 px 144 px @@ -5328,30 +5290,30 @@ 145 px - 91 px + 73 px 0 px - 91 px + 73 px false - 2288 px - 1624 px + 1880 px + 800 px 150 px - 410 px + 392 px 100 px - 410 px + 392 px 800 px - 410 px + 392 px @@ -5362,159 +5324,20 @@ - 276 - 0 - - - - - 277 - 276 - - - - - 278 - 277 - - - - 0.5 px - 0.5 px - - - 144 px - 18 px - - - 72a45164-2db7-4a0a-80de-5730034ab0e1 - EntityDeveloper.EntityFrameworkCore.EntityProperty - - - - - 279 - 277 - - - - 0.5 px - 18.5 px - - - 144 px - 18 px - - - 16897381-f09a-4d44-9232-2f33d919a58e - EntityDeveloper.EntityFrameworkCore.EntityProperty - - - - - 145 px - 37 px - - - 0 px - 37 px - - - - - 280 - 276 - - - - - 281 - 280 - - - - 144 px - 18 px - - - 1607eab9-d9af-4464-b61d-1c26f0cd3e78 - EntityDeveloper.EntityFrameworkCore.EntityRelationProperty - - - - - 282 - 280 - - - - 0 px - 18 px - - - 144 px - 18 px - - - 284baa95-8993-43d5-81f8-068c60d71d83 - EntityDeveloper.EntityFrameworkCore.EntityRelationProperty - - - - - 0 px - 36 px - - - 145 px - 55 px - - - 0 px - 55 px - - false - - - - 2079 px - 1733 px - - - 150 px - 140 px - - - 100 px - 140 px - - - 800 px - 140 px - - - - 877659ce-b722-4da3-8feb-01dd6028c715 - EntityDeveloper.EntityFrameworkCore.EntityClass - - 22 px - - - - 283 + 274 0 - 284 - 283 + 275 + 274 - 285 - 284 + 276 + 275 @@ -5532,8 +5355,8 @@ - 286 - 284 + 277 + 275 @@ -5551,8 +5374,8 @@ - 287 - 284 + 278 + 275 @@ -5570,8 +5393,8 @@ - 288 - 284 + 279 + 275 @@ -5589,8 +5412,8 @@ - 289 - 284 + 280 + 275 @@ -5608,8 +5431,8 @@ - 290 - 284 + 281 + 275 @@ -5627,8 +5450,8 @@ - 291 - 284 + 282 + 275 @@ -5646,8 +5469,8 @@ - 292 - 284 + 283 + 275 @@ -5665,8 +5488,8 @@ - 293 - 284 + 284 + 275 @@ -5684,8 +5507,8 @@ - 294 - 284 + 285 + 275 @@ -5703,8 +5526,8 @@ - 295 - 284 + 286 + 275 @@ -5722,8 +5545,8 @@ - 296 - 284 + 287 + 275 @@ -5751,14 +5574,14 @@ - 297 - 283 + 288 + 274 - 298 - 297 + 289 + 288 @@ -5772,8 +5595,8 @@ - 299 - 297 + 290 + 288 @@ -5789,6 +5612,25 @@ EntityDeveloper.EntityFrameworkCore.EntityRelationProperty + + + 291 + 288 + + + + 0 px + 36 px + + + 144 px + 18 px + + + 47428c9b-1cf2-45f6-94fb-c6c942d073e0 + EntityDeveloper.EntityFrameworkCore.EntityRelationProperty + + 0 px @@ -5796,30 +5638,30 @@ 145 px - 55 px + 73 px 0 px - 55 px + 73 px false - 2288 px - 2040 px + 1880 px + 1280 px 150 px - 320 px + 338 px 100 px - 320 px + 338 px 800 px - 320 px + 338 px @@ -5830,20 +5672,20 @@ - 300 + 292 0 - 301 - 300 + 293 + 292 - 302 - 301 + 294 + 293 @@ -5861,8 +5703,8 @@ - 303 - 301 + 295 + 293 @@ -5880,8 +5722,8 @@ - 304 - 301 + 296 + 293 @@ -5899,8 +5741,8 @@ - 305 - 301 + 297 + 293 @@ -5918,8 +5760,8 @@ - 306 - 301 + 298 + 293 @@ -5937,8 +5779,8 @@ - 307 - 301 + 299 + 293 @@ -5956,8 +5798,8 @@ - 308 - 301 + 300 + 293 @@ -5985,14 +5827,14 @@ - 309 - 300 + 301 + 292 - 310 - 309 + 302 + 301 @@ -6021,8 +5863,8 @@ - 2498 px - 506 px + 1680 px + 232 px 150 px @@ -6045,20 +5887,20 @@ - 311 + 303 0 - 312 - 311 + 304 + 303 - 313 - 312 + 305 + 304 @@ -6076,8 +5918,8 @@ - 314 - 312 + 306 + 304 @@ -6095,8 +5937,8 @@ - 315 - 312 + 307 + 304 @@ -6114,8 +5956,8 @@ - 316 - 312 + 308 + 304 @@ -6133,8 +5975,8 @@ - 317 - 312 + 309 + 304 @@ -6162,14 +6004,14 @@ - 318 - 311 + 310 + 303 - 319 - 318 + 311 + 310 @@ -6198,8 +6040,8 @@ - 1869 px - 99 px + 1440 px + 120 px 150 px @@ -6222,20 +6064,20 @@ - 320 + 312 0 - 321 - 320 + 313 + 312 - 322 - 321 + 314 + 313 @@ -6253,8 +6095,8 @@ - 323 - 321 + 315 + 313 @@ -6272,8 +6114,8 @@ - 324 - 321 + 316 + 313 @@ -6291,8 +6133,8 @@ - 325 - 321 + 317 + 313 @@ -6310,8 +6152,8 @@ - 326 - 321 + 318 + 313 @@ -6329,8 +6171,8 @@ - 327 - 321 + 319 + 313 @@ -6358,14 +6200,14 @@ - 328 - 320 + 320 + 312 - 329 - 328 + 321 + 320 @@ -6394,8 +6236,8 @@ - 1869 px - 2104 px + 1680 px + 1776 px 150 px @@ -6418,20 +6260,20 @@ - 330 + 322 0 - 331 - 330 + 323 + 322 - 332 - 331 + 324 + 323 @@ -6449,8 +6291,8 @@ - 333 - 331 + 325 + 323 @@ -6468,8 +6310,8 @@ - 334 - 331 + 326 + 323 @@ -6487,8 +6329,8 @@ - 335 - 331 + 327 + 323 @@ -6506,8 +6348,8 @@ - 336 - 331 + 328 + 323 @@ -6535,8 +6377,8 @@ - 337 - 330 + 329 + 322 @@ -6555,8 +6397,8 @@ - 2920 px - 72 px + 440 px + 200 px 150 px @@ -6579,20 +6421,20 @@ - 338 + 330 0 - 339 - 338 + 331 + 330 - 340 - 339 + 332 + 331 @@ -6610,8 +6452,8 @@ - 341 - 339 + 333 + 331 @@ -6629,8 +6471,8 @@ - 342 - 339 + 334 + 331 @@ -6648,8 +6490,8 @@ - 343 - 339 + 335 + 331 @@ -6667,8 +6509,8 @@ - 344 - 339 + 336 + 331 @@ -6696,14 +6538,14 @@ - 345 - 338 + 337 + 330 - 346 - 345 + 338 + 337 @@ -6732,8 +6574,8 @@ - 2288 px - 2619 px + 1880 px + 1920 px 150 px @@ -6756,20 +6598,20 @@ - 347 + 339 0 - 348 - 347 + 340 + 339 - 349 - 348 + 341 + 340 @@ -6787,8 +6629,8 @@ - 350 - 348 + 342 + 340 @@ -6806,8 +6648,8 @@ - 351 - 348 + 343 + 340 @@ -6835,14 +6677,14 @@ - 352 - 347 + 344 + 339 - 353 - 352 + 345 + 344 @@ -6856,8 +6698,8 @@ - 354 - 352 + 346 + 344 @@ -6890,8 +6732,8 @@ - 2288 px - 2402 px + 1880 px + 1680 px 150 px @@ -6914,49 +6756,39 @@ - 355 + 347 50 - 79 - 356 - 357 + 78 + 348 + 349 0 - 75.4154929577464 px - 0.13380281690138 px + 77.5 px + 0 px Left Right Top - 356 + 348 FloatAnchor - 75.4154929577464 px - 139.693661971831 px + 72.5 px + 0 px Left Right Bottom - 357 + 349 FloatAnchor - - - 1939.41549295775 px - 1540 px - - - 1944.41549295775 px - 1540 px - - false 6c964145-9e2a-40da-9b3b-4a2468a11c64 @@ -6965,49 +6797,39 @@ - 358 - 86 + 350 + 85 1 - 359 - 360 + 351 + 352 0 - 0.235915492957702 px - 214.268327035383 px + 0 px + 252 px Top Bottom Left - 359 + 351 FloatAnchor - 150.007042253521 px - 277.257042253521 px + 0 px + 176 px Top Bottom Right - 360 + 352 FloatAnchor - - - 2268 px - 621.268327035383 px - - - 2268 px - 612.257042253521 px - - false 5d47c83f-78c8-4173-9ea6-57e002128e7f @@ -7016,47 +6838,47 @@ - 361 - 109 + 353 + 108 50 - 362 - 363 + 354 + 355 0 - 149.897887323944 px - 203.457746478873 px + 0 px + 270.5 px Top Bottom Right - 362 + 354 FloatAnchor - -0.341549295774712 px - 10.5173280345974 px + 0 px + 47.8376865671642 px Top Bottom Left - 363 + 355 FloatAnchor - 1636 px - 1571.45774647887 px + 1290 px + 1630.5 px - 1636 px - 1570.5173280346 px + 1290 px + 1628.83764648438 px false @@ -7067,11 +6889,11 @@ - 364 - 173 - 183 - 365 - 366 + 356 + 172 + 182 + 357 + 358 0 @@ -7083,7 +6905,7 @@ Left Right Top - 365 + 357 FloatAnchor @@ -7095,7 +6917,7 @@ Left Right Bottom - 366 + 358 FloatAnchor @@ -7108,35 +6930,35 @@ - 367 - 173 - 146 - 368 - 369 + 359 + 172 + 145 + 360 + 361 0 - 75.1161971830986 px - 193.5 px + 76.5 px + 0 px Left Right Bottom - 368 + 360 FloatAnchor - 75.1161971830986 px - -0.0598591549296543 px + 73.5 px + 0 px Left Right Top - 369 + 361 FloatAnchor @@ -7149,49 +6971,39 @@ - 370 - 207 - 146 - 371 - 372 + 362 + 206 + 145 + 363 + 364 0 - 150.577464788732 px - 5.01056338028161 px + 0 px + 16.5 px Top Bottom Right - 371 + 363 FloatAnchor - -0.640845070422529 px - 238.866197183098 px + 0 px + 231.5 px Top Bottom Left - 372 + 364 FloatAnchor - - - 519.017605633803 px - 1188.01056338028 px - - - 519.017605633803 px - 1165.8661971831 px - - false c8def0da-289f-4093-8f06-d4c8c7686734 @@ -7200,35 +7012,35 @@ - 373 - 173 - 200 - 374 - 375 + 365 + 172 + 199 + 366 + 367 0 - -0.640845070422529 px - 96.7640845070423 px + 0 px + 116 px Top Bottom Left - 374 + 366 FloatAnchor - 149.919014084507 px - 69.7640845070423 px + 0 px + 70 px Top Bottom Right - 375 + 367 FloatAnchor @@ -7241,45 +7053,39 @@ - 376 - 207 - 200 - 377 - 378 + 368 + 206 + 199 + 369 + 370 0 - 98.1302816901407 px - 0.348591549295634 px + 80.5 px + 0 px Left Right Top - 377 + 369 FloatAnchor - -0.429577464788736 px - 69.7640845070423 px + 69.5 px + 0 px - Top Bottom - Left + Left Right + Bottom - 378 + 370 FloatAnchor - - - 407.130281690141 px - 770.764084507042 px - - false 8f002e41-d6a2-4466-b4ac-083741d953c7 @@ -7288,35 +7094,35 @@ - 379 - 207 - 243 - 380 - 381 + 371 + 206 + 242 + 372 + 373 0 - 8.38732394366184 px - 0.348591549295634 px + 0 px + 123 px - Left Right - Top + Top Bottom + Left - 380 + 372 FloatAnchor - 74.3873239436618 px - 212.323943661972 px + 0 px + 106 px - Left Right - Bottom + Top Bottom + Right - 381 + 373 FloatAnchor @@ -7329,35 +7135,35 @@ - 382 - 207 - 165 - 383 - 384 + 374 + 206 + 164 + 375 + 376 0 - 0.228873239436552 px - 331.348591549296 px + 0 px + 336 px Top Bottom Left - 383 + 375 FloatAnchor - 150.348591549296 px - 79.3485915492956 px + 0 px + 79 px Top Bottom Right - 384 + 376 FloatAnchor @@ -7370,35 +7176,35 @@ - 385 - 207 - 159 - 386 - 387 + 377 + 206 + 158 + 378 + 379 0 - 74.8204225352113 px - 661.183098591549 px + 0 px + 518 px - Left Right - Bottom + Top Bottom + Left - 386 + 378 FloatAnchor - 74.8204225352113 px - -0.37676056338023 px + 0 px + 61 px - Left Right - Top + Top Bottom + Right - 387 + 379 FloatAnchor @@ -7411,35 +7217,35 @@ - 388 - 207 - 109 - 389 - 390 + 380 + 206 + 108 + 381 + 382 0 - 150.577464788732 px - 495.683098591549 px + 0 px + 417 px Top Bottom Right - 389 + 381 FloatAnchor - -0.450704225352183 px - 310.683098591549 px + 0 px + 240 px Top Bottom Left - 390 + 382 FloatAnchor @@ -7452,49 +7258,39 @@ - 391 - 207 - 109 - 392 - 393 + 383 + 206 + 108 + 384 + 385 0 - 150.577464788732 px - 424.588028169014 px + 0 px + 457 px Top Bottom Right - 392 + 384 FloatAnchor - -0.450704225352183 px - 220.94014084507 px + 0 px + 280 px Top Bottom Left - 393 + 385 FloatAnchor - - - 1361.66901408451 px - 1607.58802816901 px - - - 1361.66901408451 px - 1588.94014084507 px - - false 9bffa19f-1610-417c-946e-fcb6b00b6564 @@ -7503,35 +7299,35 @@ - 394 - 207 + 386 + 206 50 - 395 - 396 + 387 + 388 0 - 150.577464788732 px - 656.521126760563 px + 0 px + 530 px Top Bottom Right - 395 + 387 FloatAnchor - -0.341549295774712 px - 308.612705277306 px + 0 px + 127.567164179104 px Top Bottom Left - 396 + 388 FloatAnchor @@ -7539,11 +7335,11 @@ 479 px - 1839.52112676056 px + 1713 px 479 px - 1868.61270527731 px + 1708.56713867188 px false @@ -7554,47 +7350,47 @@ - 397 - 207 + 389 + 206 50 - 398 - 399 + 390 + 391 0 - 150.577464788732 px - 611.066901408451 px + 0 px + 629 px Top Bottom Right - 398 + 390 FloatAnchor - -0.341549295774712 px - 210.111624101455 px + 0 px + 223.242537313433 px Top Bottom Left - 399 + 391 FloatAnchor - 1749.77816901408 px - 1794.06690140845 px + 479 px + 1812 px - 1749.77816901408 px - 1770.11162410145 px + 479 px + 1804.24255371094 px false @@ -7605,11 +7401,11 @@ - 400 - 207 + 392 + 206 31 - 401 - 402 + 393 + 394 0 @@ -7621,7 +7417,7 @@ Top Bottom Right - 401 + 393 FloatAnchor @@ -7633,22 +7429,22 @@ Top Bottom Left - 402 + 394 FloatAnchor - 1242.78873239437 px + 1040 px 1514.3485915493 px - 1242.78873239437 px - 1248.6161971831 px + 1040 px + 1269.6161971831 px - false + true 4ed7dec9-ffc6-4cba-a21e-c8f1eecc6673 EntityDeveloper.EntityFrameworkCore.EntityAssociation @@ -7656,11 +7452,11 @@ - 403 - 207 + 395 + 206 31 - 404 - 405 + 396 + 397 0 @@ -7672,7 +7468,7 @@ Top Bottom Right - 404 + 396 FloatAnchor @@ -7684,22 +7480,22 @@ Top Bottom Left - 405 + 397 FloatAnchor - 1183.3485915493 px + 1000 px 1467.72887323944 px - 1183.3485915493 px - 1158.87323943662 px + 1000 px + 1179.87323943662 px - false + true 1fce065d-7f16-45be-90ac-ec021a4e8a89 EntityDeveloper.EntityFrameworkCore.EntityAssociation @@ -7707,11 +7503,11 @@ - 406 - 207 + 398 + 206 1 - 407 - 408 + 399 + 400 0 @@ -7723,7 +7519,7 @@ Top Bottom Right - 407 + 399 FloatAnchor @@ -7735,22 +7531,22 @@ Top Bottom Left - 408 + 400 FloatAnchor - 1064.46830985915 px + 960 px 1374.48943661972 px - 1064.46830985915 px - 880.320422535211 px + 960 px + 901.320422535211 px - false + true 241235f1-88a3-400b-82fe-a3b0b3848d64 EntityDeveloper.EntityFrameworkCore.EntityAssociation @@ -7758,11 +7554,11 @@ - 409 - 207 - 86 - 410 - 411 + 401 + 206 + 85 + 402 + 403 0 @@ -7774,7 +7570,7 @@ Top Bottom Right - 410 + 402 FloatAnchor @@ -7786,26 +7582,26 @@ Left Right Top - 411 + 403 FloatAnchor - 886.147887323944 px + 880 px 1234.63028169014 px - 886.147887323944 px + 880 px 40 px - 2362.82746478873 px + 1954.82746478873 px 40 px - false + true 49b30a55-0c3a-4abd-9fde-0677e7208966 EntityDeveloper.EntityFrameworkCore.EntityAssociation @@ -7813,11 +7609,11 @@ - 412 + 404 1 31 - 413 - 414 + 405 + 406 0 @@ -7829,7 +7625,7 @@ Left Right Bottom - 413 + 405 FloatAnchor @@ -7841,7 +7637,7 @@ Left Right Top - 414 + 406 FloatAnchor @@ -7854,11 +7650,11 @@ - 415 + 407 31 - 79 - 416 - 417 + 78 + 408 + 409 0 @@ -7870,7 +7666,7 @@ Left Right Bottom - 416 + 408 FloatAnchor @@ -7882,7 +7678,7 @@ Left Right Top - 417 + 409 FloatAnchor @@ -7895,11 +7691,11 @@ - 418 - 207 + 410 + 206 1 - 419 - 420 + 411 + 412 0 @@ -7911,7 +7707,7 @@ Top Bottom Right - 419 + 411 FloatAnchor @@ -7923,22 +7719,22 @@ Top Bottom Left - 420 + 412 FloatAnchor - 1005.02816901408 px + 920 px 1327.86971830986 px - 1005.02816901408 px - 790.577464788732 px + 920 px + 811.577464788732 px - false + true f98a38b3-817e-4ee3-957e-e5e8ec4f3ec3 EntityDeveloper.EntityFrameworkCore.EntityAssociation @@ -7946,35 +7742,35 @@ - 421 - 86 - 254 - 422 - 423 + 413 + 85 + 253 + 414 + 415 0 - 74.8274647887324 px - 410.383802816901 px + 75 px + 0 px Left Right Bottom - 422 + 414 FloatAnchor - 74.8274647887324 px - 1.2359154929577 px + 75 px + 0 px Left Right Top - 423 + 415 FloatAnchor @@ -7987,388 +7783,240 @@ - 424 - 50 - 276 - 425 - 426 + 416 + 274 + 253 + 417 + 418 0 - 150.007042253521 px - 268.43463269242 px + 75 px + 0 px - Top Bottom - Right + Left Right + Top - 425 + 417 FloatAnchor - -0.552816901408278 px - 70.3908450704225 px + 75 px + 0 px - Top Bottom - Left + Left Right + Bottom - 426 + 418 FloatAnchor - - - 2039 px - 1828.43463269242 px - - - 2039 px - 1803.39084507042 px - - false - a4478423-7385-4fc1-b845-9cdb8f97ad93 + 9432ec0d-4ae7-4ca7-9334-660f926e5447 EntityDeveloper.EntityFrameworkCore.EntityAssociation - 427 - 254 - 276 - 428 - 429 + 419 + 85 + 292 + 420 + 421 0 - 0.235915492957702 px - 205.450130558633 px + 0 px + 82 px Top Bottom Left - 428 + 420 FloatAnchor - 149.795774647887 px - 70.3908450704225 px + 0 px + 130 px Top Bottom Right - 429 + 421 FloatAnchor - - - 2268 px - 1829.45013055863 px - - - 2268 px - 1803.39084507042 px - - false - ace4e055-8875-43d9-aaa7-3eadfcb29dd4 + 1e7314fb-7c02-4d4c-94ef-f2db3cd1e8fa EntityDeveloper.EntityFrameworkCore.EntityAssociation - 430 - 283 - 254 - 431 - 432 + 422 + 1 + 303 + 423 + 424 0 - 74.8274647887324 px - -0.0140845070422984 px + 75.4154929577464 px + -0.130281690140919 px Left Right Top - 431 + 423 FloatAnchor - 74.8274647887324 px - 355.545774647887 px + 75.4154929577464 px + 176.429577464789 px Left Right Bottom - 432 + 424 FloatAnchor - - - 2362.82746478873 px - 2020 px - - - 2377.82739257813 px - 2020 px - - - 2377.82739257813 px - 2054 px - - - 2362.82746478873 px - 2054 px - - false - 9432ec0d-4ae7-4ca7-9334-660f926e5447 + bb773e95-265d-45b3-9d41-aaa1a8644b9d EntityDeveloper.EntityFrameworkCore.EntityAssociation - 433 - 86 - 300 - 434 - 435 + 425 + 50 + 312 + 426 + 427 0 - 150.584507042254 px - 214.268327035383 px + 0 px + 292 px Top Bottom Right - 434 + 426 FloatAnchor - 0.0246478873241358 px - 106.257042253521 px + 0 px + 97 px Top Bottom Left - 435 + 427 FloatAnchor - - - 2458 px - 621.268327035383 px - - - 2458 px - 612.257042253521 px - - false - 1e7314fb-7c02-4d4c-94ef-f2db3cd1e8fa + 9cfc338e-a832-4397-b767-6ce1bbe63b57 EntityDeveloper.EntityFrameworkCore.EntityAssociation - 436 - 1 - 311 - 437 - 438 + 428 + 330 + 339 + 429 + 430 0 - 75.4154929577464 px - -0.130281690140919 px + 75 px + 0 px Left Right Top - 437 + 429 FloatAnchor - 75.4154929577464 px - 176.429577464789 px + 75 px + 0 px Left Right Bottom - 438 + 430 FloatAnchor false - bb773e95-265d-45b3-9d41-aaa1a8644b9d + 54d0bb3f-c24b-483f-beda-bc916c74e20d EntityDeveloper.EntityFrameworkCore.EntityAssociation - 439 - 50 - 320 - 440 - 441 + 431 + 274 + 339 + 432 + 433 0 - 113.75 px + 75 px 0 px Left Right Bottom - 440 + 432 FloatAnchor - 108.75 px + 75 px 0 px Left Right Top - 441 - FloatAnchor - - - - - - 1977.75 px - 2116 px - - - 1992.75 px - 2116 px - - - 1992.75 px - 2084 px - - - 1977.75 px - 2084 px - - - false - - 9cfc338e-a832-4397-b767-6ce1bbe63b57 - EntityDeveloper.EntityFrameworkCore.EntityAssociation - - - - - 442 - 338 - 347 - 443 - 444 - 0 - - - - - 74.8274647887324 px - 0.235915492957702 px - - Left Right - Top - - 443 - FloatAnchor - - - - - 74.8274647887324 px - 157.795774647887 px - - Left Right - Bottom - - 444 - FloatAnchor - - - - false - - 54d0bb3f-c24b-483f-beda-bc916c74e20d - EntityDeveloper.EntityFrameworkCore.EntityAssociation - - - - - 445 - 283 - 347 - 446 - 447 - 0 - - - - - 74.8274647887324 px - 301.848591549296 px - - Left Right - Bottom - - 446 - FloatAnchor - - - - - 74.8274647887324 px - -0.711267605634021 px - - Left Right - Top - - 447 + 433 FloatAnchor @@ -8381,20 +8029,20 @@ - 448 + 434 0 - 449 - 448 + 435 + 434 - 450 - 449 + 436 + 435 @@ -8412,8 +8060,8 @@ - 451 - 449 + 437 + 435 @@ -8431,8 +8079,8 @@ - 452 - 449 + 438 + 435 @@ -8450,8 +8098,8 @@ - 453 - 449 + 439 + 435 @@ -8469,8 +8117,8 @@ - 454 - 449 + 440 + 435 @@ -8488,8 +8136,8 @@ - 455 - 449 + 441 + 435 @@ -8517,14 +8165,14 @@ - 456 - 448 + 442 + 434 - 457 - 456 + 443 + 442 @@ -8553,8 +8201,8 @@ - 2488 px - 1712 px + 1672 px + 928 px 150 px @@ -8577,23 +8225,23 @@ - 458 - 254 - 448 - 459 - 460 + 444 + 253 + 434 + 445 + 446 0 0 px - 185 px + 225 px Top Bottom - Right + Left - 459 + 445 FloatAnchor @@ -8603,9 +8251,9 @@ 97 px Top Bottom - Left + Right - 460 + 446 FloatAnchor @@ -8618,20 +8266,20 @@ - 461 + 447 0 - 462 - 461 + 448 + 447 - 463 - 462 + 449 + 448 @@ -8649,8 +8297,8 @@ - 464 - 462 + 450 + 448 @@ -8668,8 +8316,8 @@ - 465 - 462 + 451 + 448 @@ -8687,8 +8335,8 @@ - 466 - 462 + 452 + 448 @@ -8706,8 +8354,8 @@ - 467 - 462 + 453 + 448 @@ -8725,8 +8373,8 @@ - 468 - 462 + 454 + 448 @@ -8754,8 +8402,8 @@ - 469 - 461 + 455 + 447 @@ -8774,8 +8422,8 @@ - 2496.73908514258 px - 148.913040776897 px + 40 px + 200 px 150 px @@ -8798,54 +8446,50 @@ - 470 - 86 + 456 + 85 50 - 471 - 472 + 457 + 458 0 - 37.5 px - 0 px + 1.76110786971799 px + 313.710723873333 px - Left Right - Bottom + None + Left - 471 - FloatAnchor + 457 + FixedAnchor - 0 px - 134 px + 132.295682089351 px + 58.4774863676901 px - Top Bottom + None Right - 472 - FloatAnchor + 458 + FixedAnchor - 2325.5 px - 855 px - - - 2034 px - 855 px + 1640 px + 593.710723873333 px - 2034 px - 1694 px + 1640 px + 1639.47748636769 px - false + true 516dd396-dfe2-49e1-97fb-d4241370e2f4 EntityDeveloper.EntityFrameworkCore.EntityAssociation @@ -8853,20 +8497,20 @@ - 473 + 459 0 - 474 - 473 + 460 + 459 - 475 - 474 + 461 + 460 @@ -8884,8 +8528,8 @@ - 476 - 474 + 462 + 460 @@ -8903,8 +8547,8 @@ - 477 - 474 + 463 + 460 @@ -8922,8 +8566,8 @@ - 478 - 474 + 464 + 460 @@ -8951,14 +8595,14 @@ - 479 - 473 + 465 + 459 - 480 - 479 + 466 + 465 @@ -8987,8 +8631,8 @@ - 1632 px - 2136 px + 1120 px + 1880 px 150 px @@ -9011,43 +8655,47 @@ - 481 + 467 50 - 473 - 482 - 483 + 459 + 468 + 469 0 0 px - 402 px + 365.305970149254 px Top Bottom Left - 482 + 468 FloatAnchor - 75 px - 0 px + 0 px + 79 px - Left Right - Top + Top Bottom + Right - 483 + 469 FloatAnchor - 1707 px - 1962 px + 1415 px + 1946.30597014925 px + + + 1415 px + 1959 px false @@ -9056,14 +8704,735 @@ EntityDeveloper.EntityFrameworkCore.EntityAssociation + + + 470 + 0 + + + + + 471 + 470 + + + + + 472 + 471 + + + + 0.5 px + 0.5 px + + + 144 px + 18 px + + + ce6a58ad-2095-47f5-ac06-5f5c55d7ebb1 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 473 + 471 + + + + 0.5 px + 18.5 px + + + 144 px + 18 px + + + a1f90e45-64c0-4ee5-ab94-f831b9e4ae9e + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 474 + 471 + + + + 0.5 px + 36.5 px + + + 144 px + 18 px + + + c4720e26-9370-4cf5-b8dd-c6d00f083246 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 475 + 471 + + + + 0.5 px + 54.5 px + + + 144 px + 18 px + + + 7f065b6b-dff0-4be4-ab05-343620674201 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 476 + 471 + + + + 0.5 px + 72.5 px + + + 144 px + 18 px + + + 8a7cfae9-ab2a-4297-bdc8-ba2c1ee15840 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 477 + 471 + + + + 0.5 px + 90.5 px + + + 144 px + 18 px + + + d795b0b3-ab7f-41ea-be07-e95a09b28343 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 478 + 471 + + + + 0.5 px + 108.5 px + + + 144 px + 18 px + + + 1eb00eb9-62d4-49ca-adb2-ac170043eb51 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 479 + 471 + + + + 0.5 px + 126.5 px + + + 144 px + 18 px + + + b8a2868e-02e8-4468-b669-de6ec9ff8614 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 480 + 471 + + + + 0.5 px + 144.5 px + + + 144 px + 18 px + + + 57b672f8-a1d9-4799-b9ce-4480138b117a + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 481 + 471 + + + + 0.5 px + 162.5 px + + + 144 px + 18 px + + + e1c8e1da-b824-49f6-a26e-b0ebc6415f60 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 482 + 471 + + + + 0.5 px + 180.5 px + + + 144 px + 18 px + + + a64efb5e-bd92-48c4-b5ef-e49422f8ddc7 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 483 + 471 + + + + 0.5 px + 198.5 px + + + 144 px + 18 px + + + 69ecd0c9-9c78-4a0c-85df-a38e57478f32 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 484 + 471 + + + + 0.5 px + 216.5 px + + + 144 px + 18 px + + + f2347512-3b7c-4664-8a87-1ffcea5d8c0a + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 485 + 471 + + + + 0.5 px + 234.5 px + + + 144 px + 18 px + + + 459c10c4-d761-4ffb-ae22-3a69bde90d94 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 486 + 471 + + + + 0.5 px + 252.5 px + + + 144 px + 18 px + + + 0ad8a140-96f7-4fe8-a394-5d8f1f9db894 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 487 + 471 + + + + 0.5 px + 270.5 px + + + 144 px + 18 px + + + 452a458c-6dcf-4106-8e89-786075b756e9 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 488 + 471 + + + + 0.5 px + 288.5 px + + + 144 px + 18 px + + + 5aadfa75-ba8d-45dd-83d3-fbb0b8972786 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 489 + 471 + + + + 0.5 px + 306.5 px + + + 144 px + 18 px + + + 390c6866-2510-4c6c-9248-0c0bcb3fefce + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 490 + 471 + + + + 0.5 px + 324.5 px + + + 144 px + 18 px + + + 931ab459-7383-4ca7-8493-04ae3a61e510 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 491 + 471 + + + + 0.5 px + 342.5 px + + + 144 px + 18 px + + + 7e83dc14-0bc7-40f9-b54b-9af281ec2e15 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 492 + 471 + + + + 0.5 px + 360.5 px + + + 144 px + 18 px + + + bc365d04-f299-47bd-afa4-a4e6ba83fb9e + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 493 + 471 + + + + 0.5 px + 378.5 px + + + 144 px + 18 px + + + af16885c-98e1-46f3-b82f-90a149140c7b + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 494 + 471 + + + + 0.5 px + 396.5 px + + + 144 px + 18 px + + + 8b278a50-d284-44b9-8070-631c777bdcfa + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 495 + 471 + + + + 0.5 px + 414.5 px + + + 144 px + 18 px + + + 0e2a6c34-bd01-4940-a0fd-4ee9afbc3da0 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 496 + 471 + + + + 0.5 px + 432.5 px + + + 144 px + 18 px + + + 9634a617-3aff-4a49-a681-1b38c96bed04 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 145 px + 451 px + + + 0 px + 451 px + + + + + 497 + 470 + + + + 0 px + 450 px + + + 145 px + 19 px + + + 0 px + 19 px + + false + + + + 48 px + 600 px + + + 150 px + 518 px + + + 100 px + 518 px + + + 800 px + 518 px + + + + 52fc09c5-6d3e-4fa6-974e-7a3ab3851447 + EntityDeveloper.EntityFrameworkCore.EntityClass + + 22 px + + + + 498 + 0 + + + + + 499 + 498 + + + + + 500 + 499 + + + + 0.5 px + 0.5 px + + + 144 px + 18 px + + + aaebf213-5120-4a80-8ac7-ed32816606a7 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 501 + 499 + + + + 0.5 px + 18.5 px + + + 144 px + 18 px + + + ca7cb144-b1d4-46cf-ada0-95eb7415c1c7 + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 502 + 499 + + + + 0.5 px + 36.5 px + + + 144 px + 18 px + + + 8f45acfe-8983-47e4-9ee9-5477026cb1cf + EntityDeveloper.EntityFrameworkCore.EntityProperty + + + + + 145 px + 55 px + + + 0 px + 55 px + + + + + 503 + 498 + + + + + 504 + 503 + + + + 144 px + 18 px + + + 297e6d52-841c-4edc-821c-b3da1641fb10 + EntityDeveloper.EntityFrameworkCore.EntityRelationProperty + + + + + 0 px + 54 px + + + 145 px + 37 px + + + 0 px + 37 px + + false + + + + 1688 px + 1384 px + + + 150 px + 140 px + + + 100 px + 140 px + + + 800 px + 140 px + + + + 5d574bbd-0f8f-47d8-a04b-8ec1b3b49c71 + EntityDeveloper.EntityFrameworkCore.EntityClass + + 22 px + + + + 505 + 274 + 498 + 506 + 507 + 0 + + + + + 0 px + 174 px + + Top Bottom + Left + + 506 + FloatAnchor + + + + + 0 px + 70 px + + Top Bottom + Right + + 507 + FloatAnchor + + + + false + + 1f0f623c-b9df-4f60-9654-704a3505e298 + EntityDeveloper.EntityFrameworkCore.EntityAssociation + + 8 px Free - 0.8 + 0.69 - 1096.2499836646 px - 1612.49997597188 px + -389.371982021742 px + 282.12560572984 px diff --git a/portal/Models/PortalModels/PortalModels.Region.cs b/portal/Models/PortalModels/PortalModels.Region.cs index a8158412..371fedcd 100644 --- a/portal/Models/PortalModels/PortalModels.Region.cs +++ b/portal/Models/PortalModels/PortalModels.Region.cs @@ -2,12 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.Role.cs b/portal/Models/PortalModels/PortalModels.Role.cs index f8cdd582..3a32ea8d 100644 --- a/portal/Models/PortalModels/PortalModels.Role.cs +++ b/portal/Models/PortalModels/PortalModels.Role.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.RoleClaim.cs b/portal/Models/PortalModels/PortalModels.RoleClaim.cs index 888e1a13..d63cb3cf 100644 --- a/portal/Models/PortalModels/PortalModels.RoleClaim.cs +++ b/portal/Models/PortalModels/PortalModels.RoleClaim.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.S3Logging.cs b/portal/Models/PortalModels/PortalModels.S3Logging.cs new file mode 100644 index 00000000..e3e5128f --- /dev/null +++ b/portal/Models/PortalModels/PortalModels.S3Logging.cs @@ -0,0 +1,111 @@ +/* +* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr +* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of +* the GNU General Public License as published by the Free Software Foundation, either version +* 3 of the License.See LICENSE for details +* +* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. +* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. +* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. +*/ +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; + +namespace PortalModels +{ + /// ksanGW 로그 + public partial class S3Logging + { + + public S3Logging() + { + OnCreated(); + } + + /// 아이디 + public virtual long Id { get; set; } + + /// 유저명 + public virtual string UserName { get; set; } + + /// 버킷명 + public virtual string BucketName { get; set; } + + /// 발생시각 + public virtual DateTime InDate { get; set; } + + /// 원격 호스트 + public virtual string RemoteHost { get; set; } + + public virtual string RequestUser { get; set; } + + /// 요청 아이디 + public virtual string RequestId { get; set; } + + /// API 명 + public virtual string Operation { get; set; } + + /// 오브젝트명 + public virtual string ObjectName { get; set; } + + /// 요청URI + public virtual string RequestUri { get; set; } + + /// 상태코드 + public virtual int StatusCode { get; set; } + + /// 에러코드 + public virtual string ErrorCode { get; set; } + + /// 응답 크기 + public virtual long ResponseLength { get; set; } + + /// 오브젝트 크기 + public virtual long ObjectLength { get; set; } + + /// 총 소요시간 + public virtual long TotalTime { get; set; } + + /// 요청 크기 + public virtual long RequestLength { get; set; } + + /// 참조 + public virtual string Rererer { get; set; } + + /// 대리인 + public virtual string UserAgent { get; set; } + + /// 버전아이디 + public virtual string VersionId { get; set; } + + /// 호스트아이디 + public virtual string HostId { get; set; } + + /// 사인 + public virtual string Sign { get; set; } + + /// SSL 그륩 + public virtual string SslGroup { get; set; } + + /// 사인 종류 + public virtual string SignType { get; set; } + + /// 엔드포인트 + public virtual string Endpoint { get; set; } + + /// TLS 버전 + public virtual string TlsVersion { get; set; } + + #region Extensibility Method Definitions + + partial void OnCreated(); + + #endregion + } + +} diff --git a/portal/Models/PortalModels/PortalModels.Server.cs b/portal/Models/PortalModels/PortalModels.Server.cs index 7a14e2cf..97487d21 100644 --- a/portal/Models/PortalModels/PortalModels.Server.cs +++ b/portal/Models/PortalModels/PortalModels.Server.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.ServerUsage.cs b/portal/Models/PortalModels/PortalModels.ServerUsage.cs index 6a20d666..788e94b7 100644 --- a/portal/Models/PortalModels/PortalModels.ServerUsage.cs +++ b/portal/Models/PortalModels/PortalModels.ServerUsage.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.Service.cs b/portal/Models/PortalModels/PortalModels.Service.cs index a1424aa0..ed7fef60 100644 --- a/portal/Models/PortalModels/PortalModels.Service.cs +++ b/portal/Models/PortalModels/PortalModels.Service.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { @@ -26,7 +31,6 @@ public Service() this.MemoryUsed = 0m; this.ThreadCount = 0; this.Vlans = new List(); - this.ServiceDisks = new List(); this.ServiceUsages = new List(); this.ServiceEventLogs = new List(); OnCreated(); @@ -98,9 +102,6 @@ public Service() /// 수정 사용자 정보 public virtual User ModUser { get; set; } - /// 서비스 디스크 정보 목록 - public virtual IList ServiceDisks { get; set; } - public virtual IList ServiceUsages { get; set; } public virtual Server Server { get; set; } diff --git a/portal/Models/PortalModels/PortalModels.ServiceConfig.cs b/portal/Models/PortalModels/PortalModels.ServiceConfig.cs index cd3824de..4b111304 100644 --- a/portal/Models/PortalModels/PortalModels.ServiceConfig.cs +++ b/portal/Models/PortalModels/PortalModels.ServiceConfig.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.ServiceDisk.cs b/portal/Models/PortalModels/PortalModels.ServiceDisk.cs deleted file mode 100644 index d1f665ac..00000000 --- a/portal/Models/PortalModels/PortalModels.ServiceDisk.cs +++ /dev/null @@ -1,67 +0,0 @@ -/* -* Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr -* KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of -* the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details -* -* 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. -* KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. -* KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. -*/ -using System; - -namespace PortalModels -{ - /// 서비스 사용 디스크 정보 - public partial class ServiceDisk - { - - public ServiceDisk() - { - OnCreated(); - } - - /// 서비스 아이디 - public virtual Guid ServiceId { get; set; } - - /// 디스크 아이디 - public virtual Guid DiskId { get; set; } - - /// 서비스 정보 - public virtual Service Service { get; set; } - - /// 디스크 정보 - public virtual Disk Disk { get; set; } - - #region Extensibility Method Definitions - - partial void OnCreated(); - - public override bool Equals(object obj) - { - ServiceDisk toCompare = obj as ServiceDisk; - if (toCompare == null) - { - return false; - } - - if (!Object.Equals(this.ServiceId, toCompare.ServiceId)) - return false; - if (!Object.Equals(this.DiskId, toCompare.DiskId)) - return false; - - return true; - } - - public override int GetHashCode() - { - int hashCode = 13; - hashCode = (hashCode * 7) + ServiceId.GetHashCode(); - hashCode = (hashCode * 7) + DiskId.GetHashCode(); - return hashCode; - } - - #endregion - } - -} diff --git a/portal/Models/PortalModels/PortalModels.ServiceEventLog.cs b/portal/Models/PortalModels/PortalModels.ServiceEventLog.cs index ba4c2f07..5659714a 100644 --- a/portal/Models/PortalModels/PortalModels.ServiceEventLog.cs +++ b/portal/Models/PortalModels/PortalModels.ServiceEventLog.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.ServiceGroup.cs b/portal/Models/PortalModels/PortalModels.ServiceGroup.cs index be8eca26..f7798fb0 100644 --- a/portal/Models/PortalModels/PortalModels.ServiceGroup.cs +++ b/portal/Models/PortalModels/PortalModels.ServiceGroup.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.ServiceNetworkInterfaceVlan.cs b/portal/Models/PortalModels/PortalModels.ServiceNetworkInterfaceVlan.cs index bf8567e9..348647ed 100644 --- a/portal/Models/PortalModels/PortalModels.ServiceNetworkInterfaceVlan.cs +++ b/portal/Models/PortalModels/PortalModels.ServiceNetworkInterfaceVlan.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.ServiceUsage.cs b/portal/Models/PortalModels/PortalModels.ServiceUsage.cs index 1be2942d..eacb3998 100644 --- a/portal/Models/PortalModels/PortalModels.ServiceUsage.cs +++ b/portal/Models/PortalModels/PortalModels.ServiceUsage.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.SystemLog.cs b/portal/Models/PortalModels/PortalModels.SystemLog.cs index 0d003518..8c356365 100644 --- a/portal/Models/PortalModels/PortalModels.SystemLog.cs +++ b/portal/Models/PortalModels/PortalModels.SystemLog.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.User.cs b/portal/Models/PortalModels/PortalModels.User.cs index 4d0cbd14..d6c292c1 100644 --- a/portal/Models/PortalModels/PortalModels.User.cs +++ b/portal/Models/PortalModels/PortalModels.User.cs @@ -2,7 +2,7 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. @@ -10,6 +10,11 @@ */ using System; using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.UserActionLog.cs b/portal/Models/PortalModels/PortalModels.UserActionLog.cs index 114c284d..213bc85e 100644 --- a/portal/Models/PortalModels/PortalModels.UserActionLog.cs +++ b/portal/Models/PortalModels/PortalModels.UserActionLog.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.UserClaim.cs b/portal/Models/PortalModels/PortalModels.UserClaim.cs index 0a6cc01d..74a4e9a2 100644 --- a/portal/Models/PortalModels/PortalModels.UserClaim.cs +++ b/portal/Models/PortalModels/PortalModels.UserClaim.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.UserDiskPool.cs b/portal/Models/PortalModels/PortalModels.UserDiskPool.cs index 5985cb16..5390d134 100644 --- a/portal/Models/PortalModels/PortalModels.UserDiskPool.cs +++ b/portal/Models/PortalModels/PortalModels.UserDiskPool.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.UserLoginHistory.cs b/portal/Models/PortalModels/PortalModels.UserLoginHistory.cs index d3237b39..0d85ba3d 100644 --- a/portal/Models/PortalModels/PortalModels.UserLoginHistory.cs +++ b/portal/Models/PortalModels/PortalModels.UserLoginHistory.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.UserRole.cs b/portal/Models/PortalModels/PortalModels.UserRole.cs index c54a0b0e..cceaf936 100644 --- a/portal/Models/PortalModels/PortalModels.UserRole.cs +++ b/portal/Models/PortalModels/PortalModels.UserRole.cs @@ -2,13 +2,19 @@ * Copyright (c) 2021 PSPACE, inc. KSAN Development Team ksan@pspace.co.kr * KSAN is a suite of free software: you can redistribute it and/or modify it under the terms of * the GNU General Public License as published by the Free Software Foundation, either version -* 3 of the License. See LICENSE for details +* 3 of the License.See LICENSE for details * * 본 프로그램 및 관련 소스코드, 문서 등 모든 자료는 있는 그대로 제공이 됩니다. * KSAN 프로젝트의 개발자 및 개발사는 이 프로그램을 사용한 결과에 따른 어떠한 책임도 지지 않습니다. * KSAN 개발팀은 사전 공지, 허락, 동의 없이 KSAN 개발에 관련된 모든 결과물에 대한 LICENSE 방식을 변경 할 권리가 있습니다. */ using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Data; +using System.Data.Common; +using System.Linq; +using System.Linq.Expressions; namespace PortalModels { diff --git a/portal/Models/PortalModels/PortalModels.csproj b/portal/Models/PortalModels/PortalModels.csproj index d58977af..6412b68a 100644 --- a/portal/Models/PortalModels/PortalModels.csproj +++ b/portal/Models/PortalModels/PortalModels.csproj @@ -41,6 +41,10 @@ True PortalModels.efml + + True + PortalModels.efml + True PortalModels.efml @@ -136,6 +140,10 @@ True PortalModels.efml + + True + PortalModels.efml + True PortalModels.efml @@ -152,7 +160,7 @@ True PortalModels.efml - + True PortalModels.efml diff --git a/portal/Models/PortalModels/PortalModels.edps b/portal/Models/PortalModels/PortalModels.edps index 234e6980..7d8f6faa 100644 --- a/portal/Models/PortalModels/PortalModels.edps +++ b/portal/Models/PortalModels/PortalModels.edps @@ -1,5 +1,5 @@  - + @@ -24,7 +24,6 @@ - @@ -35,6 +34,8 @@ + + @@ -68,14 +69,14 @@