Home

##Usage Examples##

NOTE: Must be run as root.

###Cache Monitoring Technology (CMT) and Memory B/W Monitoring (MBM) usage:###

• ####Monitor all events on cores 0 to 11:#### pqos -m all:0-11
pqos -m :0-11

• ####Monitor LLC on cores 0, 2 and 6:#### pqos -m llc:0,2,6

• ####Monitor local memory B/W on cores 0-2 and remote memory B/W on cores 3, 4 and 5:#### pqos -m "mbl:0-2;mbr:3,4,5"

• ####Monitor events on groups of cores (aggregate statistics):#### pqos -m "all:[0-11];llc:[12,13,14];mbl:[15-17,20]"

• ####Monitor all events for PID's 43523, 43589 and 3245 to 3252:#### pqos -p all:43523,43589,3245-3252

• ####Reset Monitoring:#### Reclaims in-use RMID's.
  pqos -r

####Example CMT/MBM usage scenario#### A user has a host machine running 3 guest VM's with 3 cores assigned to each guest.

• VM0 - cores 0-2
• VM1 - cores 3-5
• VM2 - cores 6-8

To monitor all events (LLC occupancy, local and remote memory B/W) run:
pqos -m "all:[0-2],[3-5],[6-8];"

The output will look similar to the table below.

CORE	IPC	MISSES	LLC[KB]	MBL[MB/s]	MBR[MB/s]
0-2	0.47	39k	5645.0	258.3	2.3
3-5	0.24	12k	2341.0	452.2	0.0
6-8	0.28	16k	112.0	12.8	0.0

---

###Cache Allocation Technology (CAT) usage:###

• ####Set COS 1 to the first 4 cache ways and COS 2 to the next 8 cache ways:#### pqos -e "llc:1=0x000f;llc:2=0x0ff0;"

• ####Associate cores 0, 2, and 6 to 10 with COS 1 and core 1 to COS 2:#### pqos -a "llc:1=0,2,6-10;llc:2=1;"

• ####Enable, disable CDP:#### pqos -S cdp-on
pqos -S cdp-off

• ####Use current CDP settings and set COS 1 code and data bitmasks:#### pqos -S cdp-any -e "llc:1d=0xfff;llc:1c=0xfff00;"

• ####Show current CAT settings:#### pqos -s

• ####Reset CAT:#### Sets all COS to default (fill into all ways) and associates all cores with COS 0.
  pqos -R

####Example CAT usage scenario#### A user has a host machine running 3 guest VM's. Each guest is assigned 3 cores and a priority.

• VM0 - cores 0-2 (P5)
• VM1 - cores 3-5 (P2)
• VM2 - cores 6-8 (P1)

As VM0 has the highest priority it will be assigned 8 exclusive LLC ways. VM1 and VM2 are relatively low priority so VM1 is assigned 6 ways and VM2 is assigned 4 ways, 2 of which will be shared.

• First, set the 3 COS bitmasks for each VM:
  pqos -e "llc:1=0x00ff;llc:2=0x3f00;llc:3=0xf000;"

• Next, associate each COS with the cores where each VM is running:
  pqos -a "llc:1=0-2;llc:2=3-5;llc:3=6-8;"

VM 0 now has exclusive access to 8 LLC ways, VM1 has exclusive access to 4 ways and shared access to 2 ways and VM2 has exclusive access to 2 ways and shared access to another 2 ways. All other cores have access to all other ways.

##FAQ##

####How do I know if my CPU supports CMT or CAT and what other features are supported?#### If your CPU supports CAT/CMT, the pqos -s -vcommand will print the current CAT configuration. The -v option will print additional log information messages containing the following detected capabilities and LLC information:

• CMT support
• CAT support
• CDP support
• CDP on/off
• Number of Classes of Service (COS)
• LLC size
• LLC way size
• Number of LLC ways
• Ways contention bitmask
• Process monitoring support
• Process monitoring supported events

The -voption can be replaced with -V to include more verbose debug log messages.

####When I try to run the utility I get the following error messages:#### WARN: Error opening file '/dev/cpu/0/msr'!
ERROR: CDP detection error!
ERROR: Fatal error encounter in CAT discovery!
ERROR: discover_capabilities() error 1
Error initializing PQoS library!

This is usually seen when the utility is run without root privileges. Another possible cause is when the msr driver has not been auto-loaded.
For some modular kernels the driver may need to be loaded manually by running:
modprobe msr

####When I try to start monitoring I get the following error messages:#### ERROR: IPC and/or LLC miss performance counters already in use!
Use -r option to start monitoring anyway.
Monitoring start error on core(s) 0, status 6

This is a warning that another application may be using PMU counters to monitor resources. To force the PQoS utility to start monitoring anyway, add the -r option to the command.
Example: pqos -r -p llc:34534 or pqos -r -m :[0-47]

####When I try to compile the library I get the following error message:#### host_pidapi.c:46:30: fatal error: linux/perf_event.h:
No such file or directory compilation terminated.

This occurs on Linux kernels that do not support Perf monitoring events. To compile the library without PID monitoring support, run:
make all NO_PID_API=y

####When I try to monitor processes/tasks I get the following error message:#### PID 1 monitoring start error,status 1

This is usually caused when the system is using a kernel that does not support PQoS Perf events such as:

• llc_occupancy - llc (LLC occupancy)
• local_mb - mbl (Local Memory B/W)
• total_mb - mbr (Used with MBL to calculate Remote Memory B/W)

NOTE: Kernel version 4.1 or newer is required for PID monitoring.

####Does CAT extend to kernel/ring 0 code?#### Q: For example, if I have a kernel module run on a particular core, will it be limited to the LLC allocation for that particular core or are the limitations just for userspace applications?
A: Currently, the utility and library offer static CAT configuration. This means cores are assigned to classes of service and this configuration doesn't change when tasks are scheduled in and out on these cores. Consequently Linux kernel threads running on this core will be also affected. Depending on the application, this may be a good or a bad thing. There is work in progress to enable CAT configuration on task basis to address this gap.

####Can the portion of cache (i.e. number of cache ways) be changed dynamically?#### Yes it can, user applications can modify COS bitmasks at runtime. However, currently CAT can only be managed through the MSR interface. The downside of the MSR interface is that the technologies can only be managed on logical core basis - no task or process tracking capability. There is work in progress to enable CAT configuration on task basis to address this gap.

####Different bit masks for different cache sizes#### One bit in a COS bitmask corresponds to one cache way. Cache way size can be calculated manually with (Total LLC size divided by number of ways). This information can also be retrieved from the utility by running pqos -s -v and looking for the following lines:
...
INFO: LLC cache size 36700160 bytes, 20 ways
INFO: LLC cache way size 1835008 bytes
...
Note: These lines will only be logged if CAT support is detected.

####How fine-grained can the cache assignment be for each COS?#### Each COS bitmask must have at least 2 contiguous bits set (i.e. 2 cache ways).
For example, for a CPU with 20MB of LLC and 20 cache ways, each way is ~1MB in size. In this case, at least 2MB of cache must be assigned to each COS, which can be incremented by 1MB thereafter.

####Is there a limit for the number of COS on my system?#### The number of classes of service is product dependent. To find out how many COS are supported by your product run pqos -s -v and look for the CAT details message:
...
INFO: CAT details: CDP support=1, CDP on=0, #COS=16, #ways=20, ways contention bit-mask 0xc0000
...
In this example, the number of COS supported by the product is 16.
Note: These lines will only be logged if CAT support is detected.