160 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3===============4Boot Interrupts5===============6 7:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com>8 9Overview10========11 12On PCI Express, interrupts are represented with either MSI or inbound13interrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a14given Core IO converts the legacy interrupt messages from PCI Express to15MSI interrupts. If the IO-APIC is disabled (via the mask bits in the16IO-APIC table entries), the messages are routed to the legacy PCH. This17in-band interrupt mechanism was traditionally necessary for systems that18did not support the IO-APIC and for boot. Intel in the past has used the19term "boot interrupts" to describe this mechanism. Further, the PCI Express20protocol describes this in-band legacy wire-interrupt INTx mechanism for21I/O devices to signal PCI-style level interrupts. The subsequent paragraphs22describe problems with the Core IO handling of INTx message routing to the23PCH and mitigation within BIOS and the OS.24 25 26Issue27=====28 29When in-band legacy INTx messages are forwarded to the PCH, they in turn30trigger a new interrupt for which the OS likely lacks a handler. When an31interrupt goes unhandled over time, they are tracked by the Linux kernel as32Spurious Interrupts. The IRQ will be disabled by the Linux kernel after it33reaches a specific count with the error "nobody cared". This disabled IRQ34now prevents valid usage by an existing interrupt which may happen to share35the IRQ line::36 37 irq 19: nobody cared (try booting with the "irqpoll" option)38 CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #139 Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/202040 Call Trace:41 42 <IRQ>43 ? dump_stack+0x46/0x5e44 ? __report_bad_irq+0x2e/0xb045 ? note_interrupt+0x242/0x29046 ? nNIKAL100_memoryRead16+0x8/0x10 [nikal]47 ? handle_irq_event_percpu+0x55/0x7048 ? handle_irq_event+0x4f/0x8049 ? handle_fasteoi_irq+0x81/0x18050 ? handle_irq+0x1c/0x3051 ? do_IRQ+0x41/0xd052 ? common_interrupt+0x84/0x8453 </IRQ>54 55 handlers:56 irq_default_primary_handler threaded usb_hcd_irq57 Disabling IRQ #1958 59 60Conditions61==========62 63The use of threaded interrupts is the most likely condition to trigger64this problem today. Threaded interrupts may not be re-enabled after the IRQ65handler wakes. These "one shot" conditions mean that the threaded interrupt66needs to keep the interrupt line masked until the threaded handler has run.67Especially when dealing with high data rate interrupts, the thread needs to68run to completion; otherwise some handlers will end up in stack overflows69since the interrupt of the issuing device is still active.70 71Affected Chipsets72=================73 74The legacy interrupt forwarding mechanism exists today in a number of75devices including but not limited to chipsets from AMD/ATI, Broadcom, and76Intel. Changes made through the mitigations below have been applied to77drivers/pci/quirks.c78 79Starting with ICX there are no longer any IO-APICs in the Core IO's80devices. IO-APIC is only in the PCH. Devices connected to the Core IO's81PCIe Root Ports will use native MSI/MSI-X mechanisms.82 83Mitigations84===========85 86The mitigations take the form of PCI quirks. The preference has been to87first identify and make use of a means to disable the routing to the PCH.88In such a case a quirk to disable boot interrupt generation can be89added. [1]_90 91Intel® 6300ESB I/O Controller Hub92 Alternate Base Address Register:93 BIE: Boot Interrupt Enable94 95 == ===========================96 0 Boot interrupt is enabled.97 1 Boot interrupt is disabled.98 == ===========================99 100Intel® Sandy Bridge through Sky Lake based Xeon servers:101 Coherent Interface Protocol Interrupt Control102 dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:103 When this bit is set. Local INTx messages received from the104 Intel® Quick Data DMA/PCI Express ports are not routed to legacy105 PCH - they are either converted into MSI via the integrated IO-APIC106 (if the IO-APIC mask bit is clear in the appropriate entries)107 or cause no further action (when mask bit is set)108 109In the absence of a way to directly disable the routing, another approach110has been to make use of PCI Interrupt pin to INTx routing tables for111purposes of redirecting the interrupt handler to the rerouted interrupt112line by default. Therefore, on chipsets where this INTx routing cannot be113disabled, the Linux kernel will reroute the valid interrupt to its legacy114interrupt. This redirection of the handler will prevent the occurrence of115the spurious interrupt detection which would ordinarily disable the IRQ116line due to excessive unhandled counts. [2]_117 118The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or119disable) the redirection of the interrupt handler to the PCH interrupt120line. The option can be overridden by either pci=ioapicreroute or121pci=noioapicreroute. [3]_122 123 124More Documentation125==================126 127There is an overview of the legacy interrupt handling in several datasheets128(6300ESB and 6700PXH below). While largely the same, it provides insight129into the evolution of its handling with chipsets.130 131Example of disabling of the boot interrupt132------------------------------------------133 134 - Intel® 6300ESB I/O Controller Hub (Document # 300641-004US)135 5.7.3 Boot Interrupt136 https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf137 138 - Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families139 Datasheet - Volume 2: Registers (Document # 330784-003)140 6.6.41 cipintrc Coherent Interface Protocol Interrupt Control141 https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf142 143Example of handler rerouting144----------------------------145 146 - Intel® 6700PXH 64-bit PCI Hub (Document # 302628)147 2.15.2 PCI Express Legacy INTx Support and Boot Interrupt148 https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf149 150 151If you have any legacy PCI interrupt questions that aren't answered, email me.152 153Cheers,154 Sean V Kelley155 sean.v.kelley@linux.intel.com156 157.. [1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/158.. [2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/159.. [3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/160