Skip to content

Latest commit

 

History

History
420 lines (344 loc) · 209 KB

README.en.md

File metadata and controls

420 lines (344 loc) · 209 KB

mugen

About mugen

mugen is a testing framework provided by openEuler community. This repo contains mugen along with testcases and testsuits for openEuler risc-v port.

Update log

  • New mugen entry function
  • Added support for setting testsuit directory and testing envrionment
  • Added python util functions for testing

How to use

Installation

Simply run bash dep_install.sh.

Set testing envrionment

  • run bash mugen.sh -c --ip $ip --password $passwd --user $user --port $port
  • Parameters
    • ip: ip address of target machine
    • user: username of target machine, default is root
    • password
    • port: ssh port of target machine, default is22
  • Configuration file is located at ./conf/env.json
{
    "NODE": [
        {
            "ID": 1,
            "LOCALTION": "local",
            "MACHINE": "physical",
            "FRAME": "aarch64",
            "NIC": "eth0",
            "MAC": "55:54:00:c8:a9:21",
            "IPV4": "192.168.0.10",
            "USER": "root",
            "PASSWORD": "openEuler12#$",
            "SSH_PORT": 22,
            "BMC_IP": "",
            "BMC_USER": "",
            "BMC_PASSWORD": ""
        },
        {
            "ID": 2,
            "LOCALTION": "remote",
            "MACHINE": "kvm",
            "FRAME": "aarch64",
            "NIC": "eth0",
            "MAC": "55:54:00:c8:a9:22",
            "IPV4": "192.168.0.11",
            "USER": "root",
            "PASSWORD": "openEuler12#$",
            "SSH_PORT": 22,
            "HOST_IP": "",
            "HOST_USER": "",
            "HOST_PASSWORD": ""
            "HOST_SSH_PORT": ""
        }
    ]
}
  • In the testcases, use NODE${id}_${LOCALTION} to use the configured environment.

Run the test

  • Run all test cases bash mugen.sh -a
  • Run a test suit bash mugen.sh -f testsuite
  • Run a test case bash mugen.sh -f testsuite -r testcase
  • Scve the testing log
bash mugen.sh -a -x 
bash mugen.sh -f testsuite -x
bash mugen.sh -f testsuite -r testcase -x

Write your test case

  • Follow the template we provided:
source ${OET_PATH}/libs/locallibs/common_lib.sh

# Load the data and config 
function config_params() {
    LOG_INFO "Start to config params of the case."

    LOG_INFO "No params need to config."

    LOG_INFO "End to config params of the case."
}

# Prepare the testing environment, eg. Install required packages
function pre_test() {
    LOG_INFO "Start to prepare the test environment."

    LOG_INFO "No pkgs need to install."

    LOG_INFO "End to prepare the test environment."
}

# Run the test
function run_test() {
    LOG_INFO "Start to run test."

    # to test ls
    ls -CZl -all
    CHECK_RESULT 0

    # To check the result
    CHECK_RESULT "$(ls / | grep -cE 'proc|usr|roor|var|sys|etc|boot|dev')" 7

    LOG_INFO "End to run test."
}

# Cleaning the testing environment
function post_test() {
    LOG_INFO "Start to restore the test environment."

    LOG_INFO "Nothing to do."

    LOG_INFO "End to restore the test environment."
}

main "$@"
  • A testcase is a shell/python script, and a return value of 0 indicates a successful test.
    • Examples: oe_test_casename_02 and oe_test_casename_03

The json file in suite2cases

  • The json file is a configuration of a test suit.
  • Note that:

Util functions provided by mugen

  • How to load and call these util functions
# bash
source ${OET_PATH}/libs/locallibs/common_lib.sh

# python
import os, sys, subprocess

LIBS_PATH = os.environ.get("OET_PATH") + "/libs/locallibs"
sys.path.append(LIBS_PATH)
import xxx
  • Util functions
    • Logging
    # bash 
LOG_INFO "$message"
LOG_WARN "$message"
LOG_DEBUG "$message"
LOG_ERROR "$message"
# python
import mugen_log
mugen_log.logging(level, message) # level:INFO,WARN,DEBUG,ERROR;message:日志输出
    • Check result
    # bash
CHECK_RESULT $1 $2 $3 $4
# $1: Return value of last command
# $2: Expected value
# $3: Comp mode, 0: ret == expected -> successful ; 1: ret != expected -> successful
# $4: Output of
    • Install/Uninstall of rpm packages
    # bash
## func 1
DNF_INSTALL "vim bc" "$node_id"
DNF_REMOVE "$node_id" "jq" "$mode"

# mode: Default 0, indicates normal removal of the package; when mode is not 0, mugen will only remove the required package.
# python
import rpm_manage
tpmfile = rpm_manage.rpm_install(pkgs, node, tmpfile)
rpm_manage.rpm_remove(node, pkgs, tmpfile)
    • Remote execution
    # bash
## func 1
SSH_CMD "$cmd" $remote_ip $remote_password $remote_user $time_out $remote_port`
## func 2
P_SSH_CMD --cmd $cmd --node $node(PORT)
P_SSH_CMD --cmd $cmd --ip $remote_ip --password $remote_password --port $remote_port --user $remote_user --timeout $timeout
# python
conn = ssh_cmd.pssh_conn(remote_ip, remote_password,remote_port,remote_user,remote_timeout)
exitcode, output = ssh_cmd.pssh_cmd(conn, cmd)

# port: default is 22
# user: default is root
# timeout: default is unlimited
    • File transfer
    # bash
## func 1
### local -> remote  
`SSH_SCP $local_path/$file $REMOTE_USER@$REMOTE_IP:$remote_path "$REMOTE_PASSWD"`
### local <- remote
`SSH_SCP $REMOTE_USER@$REMOTE_IP:$remote_path/$file $local_path "$REMOTE_PASSWD"`
## func 2
### Transfer a directory
SFTP get/put --localdir $localdir --remotedir $remotedir
### Transfer a file
SFTP get/put --localdir $localdir --remotedir $remotedir --localfile/remotefile $localfile/$remotefile
# python
### Transfer a directory
import ssh_cmd, sftp
conn = ssh_cmd.pssh_conn(remote_ip, remote_password,remote_port,remote_user,remote_timeout)
psftp_get(conn,remote_dir, local_dir)
psftp_put(local_dir=local_dir, remote_dir=remote_dir)
### Transfer a directory
import ssh_cmd, sftp
psftp_get(remote_dir, remote_file, local_dir)
psftp_put(local_dir=local_dir, local_file=local_file, remote_dir=remote_dir)


# get: get from remote
# put: put into remote
# localdir: default is local $(pwd)
# remotedir: default is remote $(pwd)
    • Get a free port
    # bash
GET_FREE_PORT $ip
# python
import free_port
free_port.find_free_port(ip)
    • Check if a port is taken
    # bash 
IS_FREE_PORT $port $ip
# python
import free_port
free_port.is_free_port(port, ip)
    • Get a test NIC
    # bash
TEST_NIC $node_id
# python
import get_test_device
get_test_nic(node_id)

# node_id: 默认为1号节点
    • Get a test Disk
    # bash
TEST_DISK $node_id
# python
import get_test_device
get_test_disk(node_id)

# node_id: default is 1
    • Sleep
    # bash
SLEEP_WAIT $wait_time $cmd
# python
import sleep_wait
sleep_wait.sleep_wait(wait_time,cmd)
    • Reboot remote
    # bash
REMOTE_REBOOT_WAIT $node_id $wait_time

How to debug your testcase

  • use export OET_PATH=$(pwd) to set OET_PATH and just run the script in their correspoding dir

About timeout

  • Default timeout is 30 min
  • If a test case costs more than 30 min，set TIMEOUT in the test case script

FAQ

  • Timeout in remote execution
    • Possible cause: ssh will wait for stdout and will not exit unless there is a signal
    • Possible solution: Redirect stdout to /dev/null, like cmd > /dev/nul 2>&1 &

Using QEMU for Testing

When testing software packages on non x86-64 processor architectures like RISC-V, you can use the qemu_test.py script to call QEMU for testing.

Preparing the Virtual Machine

Download the image of the system to be tested (.qcow2.zst file) and use the unzstd command to extract it into a .qcow2 file. Also, download the bootloader (e.g., fw_payload_oe_uboot_*.bin) and the startup script (start_vm.sh). Save all these files in the same directory.

Start the virtual machine using the startup script and clone the mugen repository into a directory inside the virtual machine (e.g., /root/mugen). Then, use the dep_install.sh script to install dependencies. If necessary, you can make extra modifications to the virtual machine, such as installing the lshw package for network-dependent testing.

Once the virtual machine is prepared, shut it down using the poweroff command, and take note of the name of the .qcow2 image file.

Preparing the Physical Machine

On the physical machine (not necessarily running openEuler), clone the mugen repository. You don't need to use dep_install.sh to install dependencies.

qemu_test.py depends on the paramiko Python library, which can be installed using pip or the package manager of your operating system. For example, on Arch Linux, you can install the python-paramiko package using pacman.

If you're conducting network-related tests, you need to configure the bridge and TAP network interfaces.

First, install the bridge-utils package and use the following command to add the br0 bridge:

# brctl addbr br0

Next, add the TAP network interface. You'll need to install the uml-utilities package and enable the tun kernel module. You can use the tapsetup.sh script to add multiple TAP interfaces:

$ sudo bash tapsetup.sh 10.198.101.1 50 br0 $(whoami)

Here, 10.198.101.1 is the IP address of the bridge (the script automatically adds /24 to the IP), 50 is the number of TAP interfaces, br0 is the name of the bridge network interface, and the last argument specifies the username allowed to use the TAP interface (here, we use $(whoami) to get the current username).

Writing the Configuration File

The qemu_test.py script uses a JSON-format configuration file. Below is an example:

{
    "workingDir": ".",                      // Working directory, where virtual machine files are stored.
    "bios": "fw_payload_oe_uboot_2304.bin", // File name of the bootloader.
    "drive": "mugen_ready.qcow2",           // File name of the prepared virtual machine image.
    "user": "root",                         // Username of the virtual machine.
    "password": "openEuler12#$",            // Password of the virtual machine.
    "threads": 4,                           // Number of testing threads.
    "cores": 4,                             // Number of cores allocated to the virtual machine.
    "memory": 4,                            // Memory capacity allocated to the virtual machine in GB.
    "mugenNative": 1,                       // Whether to use the test suite on the virtual machine.
    "detailed": 1,                          // Whether to print detailed logs on the screen.
    "addDisk": 1,                           // Number of additional disks to be added.
    "mugenDir": "/root/mugen",              // Directory where mugen is located in the virtual machine.
    "listFile": "lists/list_test",          // File containing the list of test suites to be tested.
    "generate": 1,                          // Whether to save the test suite to the physical machine.
    "addNic": 1,                            // Whether to add a network card.
    "multiMachine": 1,                      // Whether to use multiple machines for testing.
    "tap num": 50,                          // Number of TAP network cards available for use.
    "bridge ip": "10.198.101.114"           // IP address of the bridge used for testing.
}

Please note that the script cannot process inline comments in JSON files. For simplicity, here's a version without comments:

{
    "workingDir": ".", 
    "bios": "fw_payload_oe_uboot_2304.bin", 
    "drive": "mugen_ready.qcow2", 
    "user": "root", 
    "password": "openEuler12#$", 
    "threads": 4, 
    "cores": 4, 
    "memory": 4, 
    "mugenNative": 1, 
    "detailed": 1, 
    "addDisk": 1, 
    "mugenDir": "/root/mugen", 
    "listFile": "lists/list_test", 
    "generate": 1, 
    "addNic": 1, 
    "multiMachine": 1, 
    "tap num": 50, 
    "bridge ip": "10.198.101.114" 
}

Performing the Test

Use the following command to call the script and perform the test using the config.json file as the configuration:

$ python qemu_test.py -F config.json

At the start of the test, the script will poll to check if the virtual machine is running. You may encounter SSH connection-related errors during this phase, which is normal.

After the test is complete, you can find relevant information regarding the test in the following folders in the working directory:

  • suite2cases_out: Test suites being tested.

  • exec_log: Logs generated during the execution of test suites.

  • logs: Logs generated during the execution of individual test cases.

You can also skip using the configuration file and directly provide configuration via command-line arguments. Refer to the --help parameter output of the script for details.

Troubleshooting

If, during the test, the script doesn't output test logs and results but continuously outputs thread information (e.g., Thread 0 is alive) for an extended period, try connecting to the SSH port specified in the configuration file (e.g., 12055). You can also use tools like top to check if QEMU is running. If everything seems abnormal, it indicates a failure.

In such a case, you can use Ctrl+C to stop the script, check the logs directory in the working directory to identify which test suites have completed, remove the already tested test suites from the list file, and then rerun the script to start testing again.

Contribution

  1. Fork the repository
  2. Create Feat_xxx branch
  3. Commit your code
  4. Create Pull Request

Gitee Feature

  1. You can use Readme_XXX.md to support different languages, such as Readme_en.md, Readme_zh.md
  2. Gitee blog blog.gitee.com
  3. Explore open source project https://gitee.com/explore
  4. The most valuable open source project GVP
  5. The manual of Gitee https://gitee.com/help
  6. The most popular members https://gitee.com/gitee-stars/