From cd4e9432388977f158c33a98e00dcaa9e808a91c Mon Sep 17 00:00:00 2001 From: KoyamaSohei Date: Wed, 6 Mar 2024 04:22:01 +0900 Subject: [PATCH] update doc --- docs/run.rst | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/docs/run.rst b/docs/run.rst index 0018ccf..9080b6d 100644 --- a/docs/run.rst +++ b/docs/run.rst @@ -9,7 +9,6 @@ Run finchfsd * ``-c db_dir`` : specifies a database directory or a DAX device. If the directory does not exist, it will be created. * ``-s db_size`` : specifies a database size. This option is only effective when using the pmemkv and fsdax. Default is 1 GiB. -* ``-t num_threads`` : specifies the number of threads of the finchfsd. Default is 1. * ``-v log_level`` : specifies the log level (e.g. debug). Default is info. An example of a script is the following. @@ -18,27 +17,19 @@ An example of a script is the following. NUM_NODES=10 NUM_CLIENTS=$((NUM_NODES * 24)) # ppn is 24 - NUM_THREADS=8 + FINCHFSD_PPN=8 FINCH_DB_SIZE=$((1024 * 1024 * 1024 * 1024)) # 1 TiB - export UCX_IB_MLX5_DEVX=n - export UCX_IB_REG_METHODS=odp,direct export UCX_NUM_EPS=$NUM_CLIENTS - mpirun -np $NUM_NODES -hostfile /path/to/hostfile --map-by ppr:1:node:PE=$NUM_THREADS -x UCX_IB_MLX5_DEVX -x UCX_IB_REG_METHODS -x UCX_NUM_EPS finchfsd -t $NUM_THREADS -c /scr -s $FINCH_DB_SIZE -v debug & + mpirun -np $((NUM_NODES*FINCHFSD_PPN)) -hostfile /path/to/hostfile --map-by ppr:$FINCHFSD_PPN:node:PE=1 -x UCX_NUM_EPS finchfsd -c /scr -s $FINCH_DB_SIZE -v debug & sleep 5 # wait for finchfsd to start -.. warning:: - - We must set ``UCX_IB_MLX5_DEVX``, ``UCX_IB_REG_METHODS``, and ``UCX_NUM_EPS``. - ``UCX_IB_MLX5_DEVX`` and ``UCX_IB_REG_METHODS`` are for enabling the RDMA on demand paging and disabling rcache because there is a problem with it. - ``UCX_NUM_EPS`` is for setting the number of endpoints and use `DC transport `_. - When we use pmemkv backend with devdax mode, we need to create namespaces for the number of threads, by `ndctl `_. We specify the ``-c /dev/dax0.%d`` option to ``finchfsd``, then ``finchfsd`` use ``/dev/dax0.0``, ``/dev/dax0.1``, ..., ``/dev/dax0.$((NUM_THREADS - 1))``. .. code-block:: bash - mpirun -np $NUM_NODES --map-by ppr:1:node:PE=$NUM_THREADS -x UCX_IB_MLX5_DEVX -x UCX_IB_REG_METHODS -x UCX_NUM_EPS finchfsd -t $NUM_THREADS -c /dev/dax0.%d -s $FINCH_DB_SIZE -v debug & + mpirun -np $((NUM_NODES*FINCHFSD_PPN)) --map-by ppr:$FINCHFSD_PPN:node:PE=1 -x UCX_NUM_EPS finchfsd -c /dev/dax0.%d -s $FINCH_DB_SIZE -v debug & ``/tmp/finchfsd`` is the `address file` and generated on the compute node where finchfsd is started. FINCHFS client library connect to finchfsd by the address file.