84 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3=======4The TLB5=======6 7When the kernel unmaps or modified the attributes of a range of8memory, it has two choices:9 10 1. Flush the entire TLB with a two-instruction sequence. This is11 a quick operation, but it causes collateral damage: TLB entries12 from areas other than the one we are trying to flush will be13 destroyed and must be refilled later, at some cost.14 2. Use the invlpg instruction to invalidate a single page at a15 time. This could potentially cost many more instructions, but16 it is a much more precise operation, causing no collateral17 damage to other TLB entries.18 19Which method to do depends on a few things:20 21 1. The size of the flush being performed. A flush of the entire22 address space is obviously better performed by flushing the23 entire TLB than doing 2^48/PAGE_SIZE individual flushes.24 2. The contents of the TLB. If the TLB is empty, then there will25 be no collateral damage caused by doing the global flush, and26 all of the individual flush will have ended up being wasted27 work.28 3. The size of the TLB. The larger the TLB, the more collateral29 damage we do with a full flush. So, the larger the TLB, the30 more attractive an individual flush looks. Data and31 instructions have separate TLBs, as do different page sizes.32 4. The microarchitecture. The TLB has become a multi-level33 cache on modern CPUs, and the global flushes have become more34 expensive relative to single-page flushes.35 36There is obviously no way the kernel can know all these things,37especially the contents of the TLB during a given flush. The38sizes of the flush will vary greatly depending on the workload as39well. There is essentially no "right" point to choose.40 41You may be doing too many individual invalidations if you see the42invlpg instruction (or instructions _near_ it) show up high in43profiles. If you believe that individual invalidations being44called too often, you can lower the tunable::45 46 /sys/kernel/debug/x86/tlb_single_page_flush_ceiling47 48This will cause us to do the global flush for more cases.49Lowering it to 0 will disable the use of the individual flushes.50Setting it to 1 is a very conservative setting and it should51never need to be 0 under normal circumstances.52 53Despite the fact that a single individual flush on x86 is54guaranteed to flush a full 2MB [1]_, hugetlbfs always uses the full55flushes. THP is treated exactly the same as normal memory.56 57You might see invlpg inside of flush_tlb_mm_range() show up in58profiles, or you can use the trace_tlb_flush() tracepoints. to59determine how long the flush operations are taking.60 61Essentially, you are balancing the cycles you spend doing invlpg62with the cycles that you spend refilling the TLB later.63 64You can measure how expensive TLB refills are by using65performance counters and 'perf stat', like this::66 67 perf stat -e68 cpu/event=0x8,umask=0x84,name=dtlb_load_misses_walk_duration/,69 cpu/event=0x8,umask=0x82,name=dtlb_load_misses_walk_completed/,70 cpu/event=0x49,umask=0x4,name=dtlb_store_misses_walk_duration/,71 cpu/event=0x49,umask=0x2,name=dtlb_store_misses_walk_completed/,72 cpu/event=0x85,umask=0x4,name=itlb_misses_walk_duration/,73 cpu/event=0x85,umask=0x2,name=itlb_misses_walk_completed/74 75That works on an IvyBridge-era CPU (i5-3320M). Different CPUs76may have differently-named counters, but they should at least77be there in some form. You can use pmu-tools 'ocperf list'78(https://github.com/andikleen/pmu-tools) to find the right79counters for a given CPU.80 81.. [1] A footnote in Intel's SDM "4.10.4.2 Recommended Invalidation"82 says: "One execution of INVLPG is sufficient even for a page83 with size greater than 4 KBytes."84