Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement-16725][Master] Add more workflow Integration test cases in master. #16711

Open
wants to merge 63 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
9887ee6
Removed Spring-boot-starter-cache dependencies
binitshrest Sep 5, 2024
9dfe630
Removed Spring-boot-starter-cache dependencies
binitshrest Sep 5, 2024
a2f3557
Merge remote-tracking branch 'origin/dev' into dev
binitshrest Sep 5, 2024
9d833b2
Merge branch 'dev' into dev
binitshrest Sep 5, 2024
12ea1d0
Merge remote-tracking branch 'origin/dev' into dev
binitshrest Sep 5, 2024
7fd1501
Merge remote-tracking branch 'origin/dev' into dev
binitshrest Sep 6, 2024
927560b
Removed cacheable and cacheconfig of spring-boot-starter-cache
binitshrest Sep 6, 2024
8f38773
Merge branch 'dev' into dev
binitshrest Sep 6, 2024
7e6f498
Removed cacheable and cacheconfig of removed spring-boot-starter-cach…
binitshrest Sep 6, 2024
5898102
Merge remote-tracking branch 'origin/dev' into dev
binitshrest Sep 6, 2024
e4f4a8d
Removed cacheable and cacheconfig of removed spring-boot-starter-cach…
binitshrest Sep 6, 2024
8c0ae9e
Removed cacheable and cacheconfig of removed spring-boot-starter-cach…
binitshrest Sep 9, 2024
cc81bbb
Removed cacheable and cacheconfig of removed spring-boot-starter-cach…
binitshrest Sep 9, 2024
1ab0f9f
Revert "Reverting some codes that is bugging"
binitshrest Sep 15, 2024
1ac7f40
Revert "Reverting some codes that is bugging"
binitshrest Sep 16, 2024
c1ab3e9
Merge branch 'dev' into dev
SbloodyS Sep 23, 2024
5ff15a1
Merge branch 'dev' into dev
binitshrest Sep 24, 2024
996a3a3
Merge branch 'dev' into dev
SbloodyS Oct 9, 2024
11d1050
renewed monaco-editor version
binitshrest Oct 9, 2024
2d3bc99
Merge remote-tracking branch 'origin/dev' into dev
binitshrest Oct 9, 2024
11f5d16
added some changes that was removed earlier
binitshrest Oct 9, 2024
2f4f8ed
reviewed some changes
binitshrest Oct 9, 2024
a166d16
reviewed some changes and updated it
binitshrest Oct 9, 2024
346b0f0
changes made that was asked
binitshrest Oct 9, 2024
e6506ca
read the documentatation of tools->release and implemented
binitshrest Oct 10, 2024
1d11fe7
kept some files in gitignore
binitshrest Oct 10, 2024
670f1b8
changing readme file
binitshrest Oct 10, 2024
0da86b4
made changes
binitshrest Oct 11, 2024
fd3a408
made changes
binitshrest Oct 11, 2024
3b9c48a
removed extra lines
binitshrest Oct 11, 2024
de40cb0
removed log4j-api mvn repo from dol-dist>release-docs>licence
binitshrest Oct 11, 2024
0fcd236
Removed log4j-api and log4j-to-slf4j from microbench pom.xml i.e tran…
binitshrest Oct 11, 2024
b7cdd41
Merge branch 'dev' into dev
SbloodyS Oct 12, 2024
a8da591
exlcuded all the transitive dependency that has log4j-api and log4j-t…
binitshrest Oct 13, 2024
27af1dd
changes are added
binitshrest Oct 13, 2024
3d13f4a
add some changes
binitshrest Oct 13, 2024
dce0895
optimize codes
binitshrest Oct 14, 2024
fc0b7f2
changes made
binitshrest Oct 14, 2024
beb21c6
made some changes
binitshrest Oct 14, 2024
de811ee
made some changes
binitshrest Oct 14, 2024
0f52a15
made some changes
binitshrest Oct 14, 2024
99321f6
removed duplicate codes
binitshrest Oct 14, 2024
cddfa31
Update pom.xml
binitshrest Oct 14, 2024
af490e2
removed two licence file log4j-api & log4j-to-slf4j as asked
binitshrest Oct 14, 2024
0689b5c
Merge branch 'dev' into dev
ruanwenjun Oct 14, 2024
ac13c0f
Merge branch 'dev' into dev
ruanwenjun Oct 15, 2024
1b4d13a
Merge branch 'dev' into Master-testcase
binitshrest Oct 21, 2024
50b7da4
wrote switchtask testcase
binitshrest Oct 21, 2024
9153cb5
made some changes
binitshrest Oct 21, 2024
8d4eead
Merge branch 'dev' into Master-testcase
SbloodyS Oct 22, 2024
40000ab
check with NullPointerException
binitshrest Oct 26, 2024
0b01617
Merge branch 'Master-testcase' of https://github.com/binitshrest/dolp…
binitshrest Oct 26, 2024
917cfe4
checking NPE while testing
binitshrest Oct 26, 2024
cadaebc
deleted unnecessary lines
binitshrest Oct 26, 2024
38a8db2
made some changes
binitshrest Oct 26, 2024
2353ac5
made some changes
binitshrest Oct 26, 2024
45ee4f8
removed some changes
binitshrest Oct 27, 2024
fe6887f
made some changes
binitshrest Oct 27, 2024
67e5c06
wrote a testcases and debugged and got null in taskParameters
binitshrest Oct 27, 2024
7551676
made some changes
binitshrest Oct 27, 2024
f3ea196
made some changes
binitshrest Oct 27, 2024
c933ee0
made some changes
binitshrest Oct 29, 2024
2d64a9e
made some changes
binitshrest Nov 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 26 additions & 20 deletions dolphinscheduler-api/src/main/resources/application.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ server:

spring:
profiles:
active: postgresql
active: mysql
banner:
charset: UTF-8
jackson:
Expand All @@ -45,12 +45,12 @@ spring:
messages:
basename: i18n/messages
datasource:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
username: root
password: root
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false
username: dolphinscheduler_user
password: StrongPassword123!
hikari:
connection-test-query: select 1
connection-test-query: SELECT 1
pool-name: DolphinScheduler
quartz:
auto-startup: false
Expand All @@ -68,14 +68,20 @@ spring:
org.quartz.jobStore.useProperties: false
org.quartz.jobStore.misfireThreshold: 60000
org.quartz.scheduler.makeSchedulerThreadDaemon: true
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
# Set this to MySQLDelegate if using MySQL
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.clusterCheckinInterval: 5000
org.quartz.scheduler.batchTriggerAcquisitionMaxCount: 1
mvc:
pathmatch:
matching-strategy: ANT_PATH_MATCHER
static-path-pattern: /static/**
cloud.discovery.client.composite-indicator.enabled: false
cloud:
discovery:
client:
composite-indicator:
enabled: false


springdoc:
swagger-ui:
Expand Down Expand Up @@ -235,15 +241,15 @@ casdoor:
# Override by profile

---
spring:
config:
activate:
on-profile: mysql
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
username: root
password: root
quartz:
properties:
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
#spring:
# config:
# activate:
# on-profile: mysql
# datasource:
# driver-class-name: com.mysql.cj.jdbc.Driver
# url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
# username: dolphinscheduler_user
# password: StrongPassword123!
# quartz:
# properties:
# org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>compile</scope>
</dependency>
</dependencies>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,12 @@
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

@Getter
@Slf4j
public class WorkflowExecutionRunnable implements IWorkflowExecutionRunnable {

@Getter
private final IWorkflowExecuteContext workflowExecuteContext;

@Getter
private final List<IWorkflowLifecycleListener> workflowInstanceLifecycleListeners;

public WorkflowExecutionRunnable(WorkflowExecutionRunnableBuilder workflowExecutionRunnableBuilder) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ public SwitchLogicTask(IWorkflowExecutionRunnable workflowExecutionRunnable,
taskExecutionContext,
JSONUtils.parseObject(taskExecutionContext.getTaskParams(), new TypeReference<SwitchParameters>() {
}));

this.workflowExecutionRunnable = workflowExecutionRunnable;
this.taskInstance = workflowExecutionRunnable
.getWorkflowExecuteContext()
Expand Down
39 changes: 20 additions & 19 deletions dolphinscheduler-master/src/main/resources/application.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,19 @@
#
spring:
profiles:
active: postgresql
active: mysql
banner:
charset: UTF-8
jackson:
time-zone: UTC
date-format: "yyyy-MM-dd HH:mm:ss"
datasource:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
username: root
password: root
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false
username: dolphinscheduler_user
password: StrongPassword123!
hikari:
connection-test-query: select 1
connection-test-query: SELECT 1
pool-name: DolphinScheduler
quartz:
job-store-type: jdbc
Expand All @@ -49,7 +49,8 @@ spring:
org.quartz.jobStore.misfireThreshold: 60000
org.quartz.scheduler.batchTriggerAcquisitionMaxCount: 1
org.quartz.scheduler.makeSchedulerThreadDaemon: true
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
# Use StdJDBCDelegate for MySQL
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.clusterCheckinInterval: 5000
cloud.discovery.client.composite-indicator.enabled: false

Expand Down Expand Up @@ -149,15 +150,15 @@ metrics:
# Override by profile

---
spring:
config:
activate:
on-profile: mysql
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
username: root
password: root
quartz:
properties:
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
#spring:
# config:
# activate:
# on-profile: mysql
# datasource:
# driver-class-name: com.mysql.cj.jdbc.Driver
# url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
# username: dolphinscheduler_user
# password: StrongPassword123!
# quartz:
# properties:
# org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
package org.apache.dolphinscheduler.server.master.runner.task.switchtask;

import org.apache.dolphinscheduler.dao.entity.TaskInstance;

import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
import org.apache.dolphinscheduler.plugin.task.api.enums.TaskExecutionStatus;
import org.apache.dolphinscheduler.plugin.task.api.model.Property;
import org.apache.dolphinscheduler.plugin.task.api.model.SwitchResultVo;
import org.apache.dolphinscheduler.plugin.task.api.parameters.SwitchParameters;
import org.apache.dolphinscheduler.server.master.engine.graph.IWorkflowExecutionGraph;
import org.apache.dolphinscheduler.server.master.engine.graph.IWorkflowGraph;
import org.apache.dolphinscheduler.server.master.engine.task.runnable.ITaskExecutionRunnable;
import org.apache.dolphinscheduler.server.master.engine.workflow.runnable.IWorkflowExecutionRunnable;
import org.apache.dolphinscheduler.server.master.engine.workflow.runnable.WorkflowExecutionRunnable;
import org.apache.dolphinscheduler.server.master.exception.MasterTaskExecuteException;
import org.apache.dolphinscheduler.server.master.runner.IWorkflowExecuteContext;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.Mock;
import org.mockito.MockitoAnnotations;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import static org.assertj.core.api.Assertions.assertThat;

import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.BDDMockito.given;
import static org.mockito.BDDMockito.then;
import static org.mockito.Mockito.*;

class SwitchLogicTaskTest {

@Mock
private TaskExecutionContext taskExecutionContext;

@Mock
private IWorkflowGraph workflowGraph;

@Mock
private IWorkflowExecutionRunnable workflowExecutionRunnable;

private SwitchLogicTask switchLogicTask;

private SwitchParameters switchParameters;

@BeforeEach
void setUp() {
MockitoAnnotations.openMocks(this);

// Mocking WorkflowExecutionContext and Graph
IWorkflowExecuteContext mockContext = mock(IWorkflowExecuteContext.class);
IWorkflowExecutionGraph mockGraph = mock(IWorkflowExecutionGraph.class);
given(mockContext.getWorkflowExecutionGraph()).willReturn(mockGraph);

workflowExecutionRunnable = mock(WorkflowExecutionRunnable.class);
given(workflowExecutionRunnable.getWorkflowExecuteContext()).willReturn(mockContext);

// Initializing taskExecutionContext and setting expected behaviors
taskExecutionContext = mock(TaskExecutionContext.class);
given(taskExecutionContext.getTaskParams()).willReturn("{}");
given(taskExecutionContext.getPrepareParamsMap()).willReturn(new HashMap<>());

// Mocking task execution runnable and task instance
ITaskExecutionRunnable mockTaskExecutionRunnable = mock(ITaskExecutionRunnable.class);
TaskInstance mockTaskInstance = mock(TaskInstance.class);
given(mockTaskExecutionRunnable.getTaskInstance()).willReturn(mockTaskInstance);
given(mockGraph.getTaskExecutionRunnableById(anyInt())).willReturn(mockTaskExecutionRunnable);

// Mock SwitchParameters and set up a basic SwitchResult
switchParameters = mock(SwitchParameters.class);
List<SwitchResultVo> switchResultVoList = new ArrayList<>();
switchResultVoList.add(new SwitchResultVo("someCondition", 123L));
SwitchParameters.SwitchResult switchResult = new SwitchParameters.SwitchResult(switchResultVoList, 999L);
given(switchParameters.getSwitchResult()).willReturn(switchResult);

// Initializing the SwitchLogicTask
switchLogicTask = new SwitchLogicTask(workflowExecutionRunnable, taskExecutionContext);
// switchLogicTask.setSwitchParameters(switchParameters); // Ensure switchParameters is set
}


@Test
void testHandle_ShouldReturnSuccess_WhenConditionMatches() throws MasterTaskExecuteException {
// Given
List<SwitchResultVo> switchResultVoList = new ArrayList<>();
SwitchResultVo switchResultVo = new SwitchResultVo("true", 123L);
switchResultVoList.add(switchResultVo);
switchParameters.setSwitchResult(new SwitchParameters.SwitchResult(switchResultVoList, 999L));

Map<String, Property> globalParams = new HashMap<>();
given(taskExecutionContext.getPrepareParamsMap()).willReturn(globalParams);

// When
switchLogicTask.handle();

// Then
then(taskExecutionContext).should().setCurrentExecutionStatus(TaskExecutionStatus.SUCCESS);
}

@Test
void testHandle_DefaultBranch_WhenNoConditionMatches() throws MasterTaskExecuteException {
// Given
List<SwitchResultVo> switchResultVoList = new ArrayList<>();
SwitchResultVo switchResultVo = new SwitchResultVo("false", 123L);
switchResultVoList.add(switchResultVo);
switchParameters.setSwitchResult(new SwitchParameters.SwitchResult(switchResultVoList, 999L));

// When
switchLogicTask.handle();

// Then
then(taskExecutionContext).should().setCurrentExecutionStatus(TaskExecutionStatus.SUCCESS);
assertThat(switchParameters.getNextBranch()).isEqualTo(999L); // Default branch should be selected
}

// @Test
// void testHandle_ThrowsException_WhenBranchDoesNotExist() {
// // Given
// given(workflowGraph.getTaskNodeByCode(anyLong())).willReturn(null);
//
// // When/Then
// assertThatThrownBy(() -> switchLogicTask.handle())
// .isInstanceOf(MasterTaskExecuteException.class)
// .hasMessageContaining("please check the switch task configuration");
// }
//
// @Test
// void testHandle_ThrowsException_WhenDefaultBranchIsMissing() {
// // Given
// switchParameters.setSwitchResult(new SwitchParameters.SwitchResult(new ArrayList<>(), null));
//
// // When/Then
// assertThatThrownBy(() -> switchLogicTask.handle())
// .isInstanceOf(IllegalArgumentException.class)
// .hasMessageContaining("please check the switch task configuration");
// }
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

spring:
profiles:
active: h2
active: mysql
jackson:
time-zone: UTC
date-format: "yyyy-MM-dd HH:mm:ss"
Expand All @@ -26,12 +26,12 @@ spring:
location: classpath:standalone-banner.txt
sql:
init:
schema-locations: classpath:sql/dolphinscheduler_h2.sql
schema-locations: classpath:sql/dolphinscheduler_mysql.sql
datasource:
driver-class-name: org.h2.Driver
url: jdbc:h2:mem:dolphinscheduler;MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=true
username: sa
password: ""
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false
username: dolphinscheduler_user
password: StrongPassword123!
quartz:
job-store-type: jdbc
jdbc:
Expand Down
10 changes: 5 additions & 5 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -308,11 +308,11 @@
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-dao-h2</artifactId>
<version>${project.version}</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.apache.dolphinscheduler</groupId>-->
<!-- <artifactId>dolphinscheduler-dao-h2</artifactId>-->
<!-- <version>${project.version}</version>-->
<!-- </dependency>-->

<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
Expand Down
Loading