200 lines · plain
1.. SPDX-License-Identifier: GPL-2.02.. _imc:3 4===================================5IMC (In-Memory Collection Counters)6===================================7 8Anju T Sudhakar, 10 May 20199 10.. contents::11 :depth: 312 13 14Basic overview15==============16 17IMC (In-Memory collection counters) is a hardware monitoring facility that18collects large numbers of hardware performance events at Nest level (these are19on-chip but off-core), Core level and Thread level.20 21The Nest PMU counters are handled by a Nest IMC microcode which runs in the OCC22(On-Chip Controller) complex. The microcode collects the counter data and moves23the nest IMC counter data to memory.24 25The Core and Thread IMC PMU counters are handled in the core. Core level PMU26counters give us the IMC counters' data per core and thread level PMU counters27give us the IMC counters' data per CPU thread.28 29OPAL obtains the IMC PMU and supported events information from the IMC Catalog30and passes on to the kernel via the device tree. The event's information31contains:32 33- Event name34- Event Offset35- Event description36 37and possibly also:38 39- Event scale40- Event unit41 42Some PMUs may have a common scale and unit values for all their supported43events. For those cases, the scale and unit properties for those events must be44inherited from the PMU.45 46The event offset in the memory is where the counter data gets accumulated.47 48IMC catalog is available at:49 https://github.com/open-power/ima-catalog50 51The kernel discovers the IMC counters information in the device tree at the52`imc-counters` device node which has a compatible field53`ibm,opal-in-memory-counters`. From the device tree, the kernel parses the PMUs54and their event's information and register the PMU and its attributes in the55kernel.56 57IMC example usage58=================59 60.. code-block:: sh61 62 # perf list63 [...]64 nest_mcs01/PM_MCS01_64B_RD_DISP_PORT01/ [Kernel PMU event]65 nest_mcs01/PM_MCS01_64B_RD_DISP_PORT23/ [Kernel PMU event]66 [...]67 core_imc/CPM_0THRD_NON_IDLE_PCYC/ [Kernel PMU event]68 core_imc/CPM_1THRD_NON_IDLE_INST/ [Kernel PMU event]69 [...]70 thread_imc/CPM_0THRD_NON_IDLE_PCYC/ [Kernel PMU event]71 thread_imc/CPM_1THRD_NON_IDLE_INST/ [Kernel PMU event]72 73To see per chip data for nest_mcs0/PM_MCS_DOWN_128B_DATA_XFER_MC0/:74 75.. code-block:: sh76 77 # ./perf stat -e "nest_mcs01/PM_MCS01_64B_WR_DISP_PORT01/" -a --per-socket78 79To see non-idle instructions for core 0:80 81.. code-block:: sh82 83 # ./perf stat -e "core_imc/CPM_NON_IDLE_INST/" -C 0 -I 100084 85To see non-idle instructions for a "make":86 87.. code-block:: sh88 89 # ./perf stat -e "thread_imc/CPM_NON_IDLE_PCYC/" make90 91 92IMC Trace-mode93===============94 95POWER9 supports two modes for IMC which are the Accumulation mode and Trace96mode. In Accumulation mode, event counts are accumulated in system Memory.97Hypervisor then reads the posted counts periodically or when requested. In IMC98Trace mode, the 64 bit trace SCOM value is initialized with the event99information. The CPMCxSEL and CPMC_LOAD in the trace SCOM, specifies the event100to be monitored and the sampling duration. On each overflow in the CPMCxSEL,101hardware snapshots the program counter along with event counts and writes into102memory pointed by LDBAR.103 104LDBAR is a 64 bit special purpose per thread register, it has bits to indicate105whether hardware is configured for accumulation or trace mode.106 107LDBAR Register Layout108---------------------109 110 +-------+----------------------+111 | 0 | Enable/Disable |112 +-------+----------------------+113 | 1 | 0: Accumulation Mode |114 | +----------------------+115 | | 1: Trace Mode |116 +-------+----------------------+117 | 2:3 | Reserved |118 +-------+----------------------+119 | 4-6 | PB scope |120 +-------+----------------------+121 | 7 | Reserved |122 +-------+----------------------+123 | 8:50 | Counter Address |124 +-------+----------------------+125 | 51:63 | Reserved |126 +-------+----------------------+127 128TRACE_IMC_SCOM bit representation129---------------------------------130 131 +-------+------------+132 | 0:1 | SAMPSEL |133 +-------+------------+134 | 2:33 | CPMC_LOAD |135 +-------+------------+136 | 34:40 | CPMC1SEL |137 +-------+------------+138 | 41:47 | CPMC2SEL |139 +-------+------------+140 | 48:50 | BUFFERSIZE |141 +-------+------------+142 | 51:63 | RESERVED |143 +-------+------------+144 145CPMC_LOAD contains the sampling duration. SAMPSEL and CPMCxSEL determines the146event to count. BUFFERSIZE indicates the memory range. On each overflow,147hardware snapshots the program counter along with event counts and updates the148memory and reloads the CMPC_LOAD value for the next sampling duration. IMC149hardware does not support exceptions, so it quietly wraps around if memory150buffer reaches the end.151 152*Currently the event monitored for trace-mode is fixed as cycle.*153 154Trace IMC example usage155=======================156 157.. code-block:: sh158 159 # perf list160 [....]161 trace_imc/trace_cycles/ [Kernel PMU event]162 163To record an application/process with trace-imc event:164 165.. code-block:: sh166 167 # perf record -e trace_imc/trace_cycles/ yes > /dev/null168 [ perf record: Woken up 1 times to write data ]169 [ perf record: Captured and wrote 0.012 MB perf.data (21 samples) ]170 171The `perf.data` generated, can be read using perf report.172 173Benefits of using IMC trace-mode174================================175 176PMI (Performance Monitoring Interrupts) interrupt handling is avoided, since IMC177trace mode snapshots the program counter and updates to the memory. And this178also provide a way for the operating system to do instruction sampling in real179time without PMI processing overhead.180 181Performance data using `perf top` with and without trace-imc event.182 183PMI interrupts count when `perf top` command is executed without trace-imc event.184 185.. code-block:: sh186 187 # grep PMI /proc/interrupts188 PMI: 0 0 0 0 Performance monitoring interrupts189 # ./perf top190 ...191 # grep PMI /proc/interrupts192 PMI: 39735 8710 17338 17801 Performance monitoring interrupts193 # ./perf top -e trace_imc/trace_cycles/194 ...195 # grep PMI /proc/interrupts196 PMI: 39735 8710 17338 17801 Performance monitoring interrupts197 198 199That is, the PMI interrupt counts do not increment when using the `trace_imc` event.200