[DSIP-26][Audit log] Audit log improvement design

[qingwli]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Motivation

Record user operation logs like create, update, delete, run, and stop jobs, etc. Easy to track item state.

Design Detail

1. Previous Design

#5822

1. Realize Method

• Plan a: publish and subscribe mode . After user completes the modification operation, the operation behavior information is published as the form of message
• Plan B: implementation using AOP

#5822 (comment)

AOP can be used to get the relevant operation records in the form of log after the end of user operation, but it is not convenient to judge whether the operation is successful.

If use the publish subscribe mode , a message can be sent to the subscriber after the user's operation is successful.

If only need to record the successfully configuration modification, it will be better to use publish and subscribe mode.

Previous design choose public subscribe mode. The reason why is just want to record success operation. And AOP can support this function.

POC:

private void saveLog(ProceedingJoinPoint joinPoint, Object object) {

        Result result = (Result)object;

        // We can judge by this code. 0 means the api is successful, other codes means we can skip this request.
        System.out.println(result.getCode());
}

Publish and subscribe mode has high code intrusive, we need to add log method anywhere. So we thought AOP it's more better.

2. Schema Design

User Name	Audit Type	Object Name	Operation Type	Time
admin	PROJECT	NewProject	CREATE	2023-12-28 10:40:23
admin	USER	NewUser	CREATE	2023-12-28 10:40:23

2. Deficiencies

The recorded operation granularity is too rough. Not clear enough to show the level like workflow and project.

1. Classification

Project

• Project (Create, Update, Delete)
• Workflow (Create, Update, Delete, Import, Export, Copy, Start, Online, Offline)
• Workflow Instance (Edit, Rerun, Stop, Kill, Pause)
• Task (Create, Update, Delete, Move, Switch version, Delete version)
• Task Instance (Force success)
• Schedule(Create, Update, Delete, Online)

Resource

• Floder (Create, Delete, Edit, Rename)
• File (Create, Delete, ReUpload, Edit, Rename, Upload)
• UDF Floder (Create, Delete, Edit)
• UDF (Upload, Edit, Delete)
• UDF Function (Create, Edit, Delete)
• Task Group (Create, Update, Switch status)

Datasource

• Datasource (Create, Update, Delete)

Security

• Tenant (Create, Update, Delete)
• User (Create, Update, Delete, Authorize)
• Alarm Group (Create, Update, Delete)
• Alarm Instance (Create, Update, Delete)
• Worker Group (Create, Update, Delete)
• Yarn Queue (Create, Update)
• Environment (Create, Update, Delete)
• Cluster (Create, Update, Delete)
• K8s namespace (Create, Update)
• Token (Create, Update, Delete)
• ......

2. Extract object

We use mutili level of object like:

• Level 1: Project.

• Level 2: Workflow.

• Level 3: Workflow Instance.

• Level 1: Resource.

• Level 2: Folder.

• Level 3: File.

Demo:

• Object Type
  • Project
  • Resource
  • Datasource
  • Security
• Object Id/Enum
  • Workflow
  • Folder
• Detail Id/Enum
  • Workflow instance
  • File

3. Extract operation type

We can extract operation type like :

• Create
• Update
• Delete
• Online
• Offline
• ......

4. New Schema Design

We have two enum defined AuditObjectType and AuditOperationType

AuditObjectType

    PROJECT("Project", null),
    PROCESS("Process", PROJECT),
    PROCESS_INSTANCE("Process instance", PROCESS),
    TASK("Task", PROCESS),
    TASK_INSTANCE("Task instance", TASK),
    SCHEDULE("Schedule", PROCESS),
		......
      
    RESOURCE("Resource", null),
    FOLDER("Folder", RESOURCE),
		......
      
    DATASOURCE("Datasource", null),

    SECURITY("Security", null),
    TENANT("Tenant", SECURITY),
    USER("User", SECURITY),  
    ......
      
    private final String name;
    private final AuditObjectType parentType;
    private final List<AuditObjectType> child = new ArrayList<>();

ParentType shows the hierarchical relationship used for UI search like we can search all project and the object belongs to the project log.

AuditOperationType

    CREATE("Create"),
    UPDATE("Update"),
    BATCH_DELETE("Batch Delete"),
		......
      
    RELEASE("Release"),
    ONLINE("Online"),
		......

    SWITCH_STATUS("Switch status"),
    SWITCH_VERSION("Switch version"),
    DELETE_VERSION("Delete version"),
    ......
      
    private final String name;

Shows the operator about the object.

AuditType

    PROJECT_CREATE(PROJECT, CREATE, ProjectOperatorImpl.class, new String[]{}, new String[]{"code"}),
    PROJECT_UPDATE(PROJECT, UPDATE, ProjectOperatorImpl.class, new String[]{}, new String[]{"code"}),
    PROJECT_DELETE(PROJECT, DELETE, ProjectOperatorImpl.class, new String[]{"code"}, new String[]{}),
 	  ......

    SCHEDULE_CREATE(SCHEDULE, CREATE, ScheduleOperatorImpl.class, new String[]{"processDefinitionCode"},
            new String[]{"id"}),
    SCHEDULE_UPDATE(SCHEDULE, UPDATE, ScheduleOperatorImpl.class, new String[]{"id"}, new String[]{}),
    SCHEDULE_ONLINE(SCHEDULE, ONLINE, ScheduleOperatorImpl.class, new String[]{"id"}, new String[]{}),
 	  ......
   
    TOKEN_CREATE(TOKEN, CREATE, TokenOperatorImpl.class, new String[]{}, new String[]{"userId"}),
    TOKEN_UPDATE(TOKEN, UPDATE, TokenOperatorImpl.class, new String[]{}, new String[]{"userId"}),
    TOKEN_DELETE(TOKEN, DELETE, TokenOperatorImpl.class, new String[]{"id"}, new String[]{}),
    ......
      
    private final Class<? extends Operator> operatorClass;
    private final AuditObjectType auditObjectType;
    private final AuditOperationType auditOperationType;

    /**
     * The names of the fields in the API request to be recorded.
     * Represents an array of key-value pairs, e.g., ["id", "status"].
     */
    private final String[] requestParamName;

    /**
     * The names of the fields in the returned object to be recorded.
     * Represents an array of field names, e.g., ["id", "code"].
     * Specify the field names to record from the returned object.
     */
    private final String[] returnObjectFieldName;

AuditType is the combo show the operator to the object. All audit type binds a operatorImpl service. requestParamName string array means we need get some information from the request like delete,when this request finished we can not get this object from db, so we need record this id and name before execution, returnObjectFieldName string array means we need get somw information from the return object like after object insert we can get the id.

5.Base operator

All audit type have some common method to record log, like get some information from requestParamName and get some information from returnObjectFieldName. Some common method code in base operator, other special code in other operatorImpl.

3. UI design

Search field:

• User

• Object Type(Project, Security)

• Object Name

• Operation Type(Create, Delete, Update)

User Name	Object Type	Object Name	Operation Type	Detail	Time
admin	PROJECT	ds-project	Create	v-project	2023-12-28 10:40:23
admin	USER	NewUser	Delete		2023-12-28 10:40:23
admin	Workflow	ds-workflow	Create	ds-workflow	2023-12-28 10:40:23
NewUser	Workflow-instance	Workflow-instance-1	Run		2023-12-28 10:40:23
NewUser	Workflow	ds-workflow	Update	add new task...	2023-12-28 10:40:23

Database Design

CREATE TABLE `t_ds_audit_log` (
  `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT'key',
  `user_id` int(11) NOT NULL COMMENT 'user id',
  `object_id` bigint(20) DEFAULT NULL COMMENT 'object id',
  `object_name` varchar(100) DEFAULT NULL COMMENT 'object id',
  `object_type` varchar(100) NOT NULL COMMENT 'object type',
  `operation_type` varchar(100) NOT NULL COMMENT 'operation type',
  `description` varchar(100) DEFAULT NULL COMMENT 'api description',
  `latency` int(11) DEFAULT NULL COMMENT 'api cost milliseconds',
  `detail` varchar(100) DEFAULT NULL COMMENT 'object change detail',
  `time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'operation time',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT= 1 DEFAULT CHARSET=utf8 COLLATE = utf8_bin;

Compatibility, Deprecation, and Migration Plan

Compatible with current system, aspect all api method, but just create, delete, method will into record log logic, other query method will keep same. Just record log not change anything about api request.

Pre design when user request api

Now design

Test Plan

Test locally and pass UT.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[qingwli]

PTAL @caishunfeng @EricGao888 @Radeity @ruanwenjun @SbloodyS

[qingwli]

Table data

[qingwli]

Search data, you can multi choose object type

[qingwli]

Search data about operation type

[rickchengx]

Search data about operation type

@qingwli if user deletes 2 process defintions in a single batch-delete operation, will there be 2 audit records in the table? or just 1 record with 2 objects

[qingwli]

Search data about operation type

@qingwli if user deletes 2 process defintions in a single batch-delete operation, will there be 2 audit records in the table? or just 1 record with 2 objects

Will record in two lines

[qingwli]

@rickchengx

[ruanwenjun]

Compatibility, Deprecation, and Migration Plan

Compatible with current system, aspect all api method, but just create, delete, method will into record log logic, other query method will keep same. Just record log not change anything about api request.

It seems is not compatible with current system, since you need to change the schema in database.

[qingwli]

Compatibility, Deprecation, and Migration Plan

Compatible with current system, aspect all api method, but just create, delete, method will into record log logic, other query method will keep same. Just record log not change anything about api request.

It seems is not compatible with current system, since you need to change the schema in database.

Yes, I changed the schema, but this table is never used before, just defined with no data