Determining The Interrupt Line For A Particular PCI-E Slot

Hi debuggers, this is Graham McIntyre again. These days I’m working more closely with hardware so I thought I’d share some hardware related debugging tips.  I recently debugged an issue where a PCI-E storage device failed to work after hot swapping it from one slot to another slot on the system without rebooting.  We determined the issue was due to the device not receiving interrupts once it was moved.   So in the process I learned how line based interrupts are routed to a particular PCI slot.    Interrupt routing is quite a hefty subject, but here’s one example of how to determine what the expected interrupt line is for a particular PCI-E slot using a live kernel debug.

 

There are two ways the routing can be defined in the ACPI tables:

  1. Static routing (most common for APIC systems)
  2. Link Node routing (most common for PIC systems)

 

Since APIC is much more common, I am focusing on method 1 for static routing. Though, it is legal to use Link Node routing with IOAPICs, it’s not common, so I am omitting how to parse that.  This is also specifically for devices that use physical line based interrupts (LBI), not Message Signaled Interrupts (MSI).

 

Here is the general method for determining the static routing IRQ for a particular device:

  1. Locate the devstack for the device, and determine its parent devices in the PCI hierarchy. (!pcitree)
  2. Determine the interrupt pin which the device uses
  3. Walk the parent devices to find the closest PCI Routing Table (_PRT) which will describe the mapping of interrupt pin to IRQ.
    1. If the parent device does not have a _PRT, then swizzle the pin, since the pin number can change when moving to the upstream side of the PCI bridge (you may end up swizzling the pin several times).  We will discuss how to swizzle the pin number later in this article.
    2. If the parent device has a _PRT, then move to the next step
  4. Convert the IntPin number from PCI to ACPI numbering
  5. Parse the _PRT method to find the static routing table
  6. Find the routing entry which represents our IntPin

 

Here’s the in-depth steps, along with an example:

Step 1: Locate the devstack for the device, and determine its parent devices in the PCI hierarchy.

 

To determine this, use !pcitree to dump the PCI hierarchy. Then locate your device by ven/dev ID.  You could also use !devnode to dump the hierarchy.

 

The way !pcitree shows the hierarchy may be a little confusing.  When it encounters a PCI bridge, it dumps the child buses under the bridge. The indenting tells you what bus a device is on. A device is always indented one level from the entry of the parent bus.  In my case, I know the device I'm interested in is VEN FEFE DEV 1550.

kd> !pcitree

Bus 0x0 (FDO Ext fffffa80053efe00)

  (d=0,  f=0) 80863406 devext 0xfffffa80054d51b0 devstack 0xfffffa80054d5060 0600 Bridge/HOST to PCI

  (d=1,  f=0) 80863408 devext 0xfffffa80054d9b70 devstack 0xfffffa80054d9a20 0604 Bridge/PCI to PCI

  Bus 0x1 (FDO Ext fffffa80054e8680)

    (d=0,  f=0) 14e41639 devext 0xfffffa80051b91b0 devstack 0xfffffa80051b9060 0200 Network Controller/Ethernet

    (d=0,  f=1) 14e41639 devext 0xfffffa80051ba1b0 devstack 0xfffffa80051ba060 0200 Network Controller/Ethernet

  (d=3,  f=0) 8086340a devext 0xfffffa80054dab70 devstack 0xfffffa80054daa20 0604 Bridge/PCI to PCI

  Bus 0x2 (FDO Ext fffffa80054e9460)

    (d=0,  f=0) 14e41639 devext 0xfffffa80051bcb70 devstack 0xfffffa80051bca20 0200 Network Controller/Ethernet

    (d=0,  f=1) 14e41639 devext 0xfffffa80051cab70 devstack 0xfffffa80051caa20 0200 Network Controller/Ethernet

  (d=4,  f=0) 8086340b devext 0xfffffa80054dbb70 devstack 0xfffffa80054dba20 0604 Bridge/PCI to PCI

  Bus 0x3 (FDO Ext fffffa80054ec190)

    (d=0,  f=0) 10000079 devext 0xfffffa80051cd1b0 devstack 0xfffffa80051cd060 0104 Mass Storage Controller/RAID

  (d=5,  f=0) 8086340c devext 0xfffffa80054dcb70 devstack 0xfffffa80054dca20 0604 Bridge/PCI to PCI

  Bus 0x4 (FDO Ext fffffa80054ede00)

    No devices have been enumerated on this bus.

  (d=6,  f=0) 8086340d devext 0xfffffa80054ddb70 devstack 0xfffffa80054dda20 0604 Bridge/PCI to PCI

  Bus 0x5 (FDO Ext fffffa80054ee9c0)

    No devices have been enumerated on this bus.

  (d=7, f=0) 8086340e devext 0xfffffa80054deb70 devstack 0xfffffa80054dea20 0604 Bridge/PCI to PCI << Root Port

  Bus 0x6 (FDO Ext fffffa80054f1190)

    (d=0, f=0) abcd8632 devext 0xfffffa80051d91b0 devstack 0xfffffa80051d9060 0604 Bridge/PCI to PCI << Upstream switch port

    Bus 0x7 (FDO Ext fffffa80051cd850)

      (d=4,  f=0) abcd8632 devext 0xfffffa80051d71b0 devstack 0xfffffa80051d7060 0604 Bridge/PCI to PCI

      Bus 0x8 (FDO Ext fffffa8006f44ac0)

        No devices have been enumerated on this bus.

      (d=5,  f=0) abcd8632 devext 0xfffffa80058d6a10 devstack 0xfffffa80058d68c0 0604 Bridge/PCI to PCI

      Bus 0x9 (FDO Ext fffffa80051ba850)

        No devices have been enumerated on this bus.

      (d=6, f=0) abcd8632 devext 0xfffffa8007075b70 devstack 0xfffffa8007075a20 0604 Bridge/PCI to PCI << Parent PDO (Downstream Switch Port)

      Bus 0xa (FDO Ext fffffa8007312b60)

(d=0, f=0) fefe1550 devext 0xfffffa8006f67b70 devstack 0xfffffa8006f67a20 0180 Mass Storage Controller/'Other' << Device

      (d=7,  f=0) abcd8632 devext 0xfffffa80051e5b70 devstack 0xfffffa80051e5a20 0604 Bridge/PCI to PCI

      Bus 0xb (FDO Ext fffffa80052d2e00)

        No devices have been enumerated on this bus.

  (d=14, f=0) 8086342e devext 0xfffffa80054dfb70 devstack 0xfffffa80054dfa20 0800 Base System Device/Interrupt Controller

  (d=14, f=1) 80863422 devext 0xfffffa80054e0b70 devstack 0xfffffa80054e0a20 0800 Base System Device/Interrupt Controller

  (d=14, f=2) 80863423 devext 0xfffffa80054e1b70 devstack 0xfffffa80054e1a20 0800 Base System Device/Interrupt Controller

  (d=1a, f=0) 80862937 devext 0xfffffa80054e2b70 devstack 0xfffffa80054e2a20 0c03 Serial Bus Controller/USB

  (d=1a, f=1) 80862938 devext 0xfffffa80054e31b0 devstack 0xfffffa80054e3060 0c03 Serial Bus Controller/USB

  (d=1a, f=7) 8086293c devext 0xfffffa80054e3b70 devstack 0xfffffa80054e3a20 0c03 Serial Bus Controller/USB

  (d=1d, f=0) 80862934 devext 0xfffffa80054e41b0 devstack 0xfffffa80054e4060 0c03 Serial Bus Controller/USB

  (d=1d, f=1) 80862935 devext 0xfffffa80054e4b70 devstack 0xfffffa80054e4a20 0c03 Serial Bus Controller/USB

  (d=1d, f=7) 8086293a devext 0xfffffa80054e51b0 devstack 0xfffffa80054e5060 0c03 Serial Bus Controller/USB

  (d=1e, f=0) 8086244e devext 0xfffffa80054e5b70 devstack 0xfffffa80054e5a20 0604 Bridge/PCI to PCI

  Bus 0xc (FDO Ext fffffa80054f2e00)

    (d=3,  f=0) 102b0532 devext 0xfffffa80051d51b0 devstack 0xfffffa80051d5060 0300 Display Controller/VGA

  (d=1f, f=0) 80862918 devext 0xfffffa80054e61b0 devstack 0xfffffa80054e6060 0601 Bridge/PCI to ISA

  (d=1f, f=2) 80862921 devext 0xfffffa80054e6b70 devstack 0xfffffa80054e6a20 0101 Mass Storage Controller/IDE

Total PCI Root busses processed = 1

Total PCI Segments processed = 1

 

To recap the devices in the tree (Bus,Device,Function):

(0,7,0) : Root Port, PCI-PCI Bridge (devstack 0xfffffa80054dea20)

    (6,0,0) : Upstream Switch Port (devstack 0xfffffa80051d9060)

        (7,6,0) : Downstream Switch Port (the PDO for the slot) (devstack 0xfffffa8007075a20)

(a,0,0) : Device (devstack 0xfffffa8006f67a20)

 

I scanned the output looking for my ven/dev ID, and found it at Bus A, Device 0, Function 0.

 

Step 2: Determine which interrupt pin the device uses.

 

For this step, you can use !pci to dump the PCI config space for the device. The output will show you the interrupt pin the device uses, labeled as IntPin.

!pci 1 a 0 0

PCI Bus 10

00:0  FEFE:1550.01  Cmd[0007:imb...]  Sts[0018:c....]  Device  SubID:1344:1008  Other mass storage controller

      cf8:800a0000  IntPin:1  IntLine:2e  Rom:0  cis:0  cap:40

      MEM[2]:df5fd000  MEM[3]:df5fc000  IO[4]:cff1       MEM[5]:df5fe000 

 

So our IntPin is 1.

 

Step 3: Walk the parent devices to find the closest PCI Routing Table (_PRT) which will describe the mapping of interrupt pin to IRQ.

 

Now, we will traverse the parent PCI devnodes until we find a PCI bridge which has an associated ACPI object with a _PRT method. This may be the root port, or an integrated bridge.

  1. Start by running !devstack on the parent.  We can determine the parent device using the indentations of the !pcitree output.
  2. If the devstack shows an ACPI filter driver, then dump the filter using !acpikd.acpiext to find the associated AcpiObject
  3. Dump the ACPI object and its children to see if it has a _PRT method defined
    1. If it does not have a _PRT, then you need to swizzle the Interrupt Pin to find what the pin number will be on the upstream side of the bridge
      1. We have to use a method called “swizzling” because the pin may become a different pin on the upstream side of the bridge. The way to calculate the pin is:
        1. IntPin = ((((IntPin -1) + DeviceNumber) % 4) +1)
      2. Where IntPin is the current IntPin value, and DeviceNumber the device number of the device you’re swizzling.
      3. You will start with the IntPin value from !pci output of the device itself. If you need to swizzle multiple times, you take the result of the previous swizzle as the input to the next swizzle
      4. The device number for the first time will be the device number of the target device, and subsequent times will be the device number of the parent device you’re swizzling.
    2. If it does have a _PRT, then move onto Step 4.

 

Example:

First, we’ll swizzle the pin of the device itself (a,0,0).  The IntPin is 1 so:

                IntPin = ((((1-1)+0) % 4) +1)    << The Swizzled Pin is still IntPin 1

 

Next, I dumped the parent device (7,6,0), !devstack  0xfffffa8007075a20. It didn’t have an ACPI filter driver on the stack. So I need to swizzle the pin.

      IntPin = ((((1-1)+6) % 4) +1)    << The Swizzled Pin is now 3

 

I now dump the next parent up, (6,0,0), !devstack 0xfffffa80051d9060. It also didn’t have an ACPI filter driver on the stack so I need to swizzle the pin again.

      IntPin = ((((3-1)+0) % 4) +1)    << The Swizzled Pin is still 3

 

I am now at the root port. The first devstack which has a _PRT method in my case is the root port.

kd> !devstack 0xfffffa80054dea20

  !DevObj   !DrvObj            !DevExt   ObjectName

  fffffa80054f1040  \Driver\pci        fffffa80054f1190 

  fffffa80054e5800  \Driver\ACPI       fffffa80051c1510 << Has an ACPI filter driver in the devstack

> fffffa80054dea20  \Driver\pci        fffffa80054deb70  NTPNP_PCI0006

!DevNode fffffa80054e1750 :

  DeviceInst is "PCI\VEN_8086&DEV_340E&SUBSYS_02351028&REV_13\3&33fd14ca&0&38"

  ServiceName is "pci"

kd> !acpikd.acpiext fffffa80051c1510

ACPI!DEVICE_EXTENSION fffffa80051c1510 - 70000

  DevObject  fffffa80054e5800  PhysObject  fffffa80054dea20  NextObject   fffffa80054dea20

  AcpiObject fffffa80052e3890 ParentExt   fffffa80051c07d0

  PnpState   Started   OldPnpState Stopped

  Dispatch   fffff880011cbb50

  RefCounts  4-Device 1-Irp 0-Hiber 0-Wake

  State      D0       

  SxD Table  S0->D0 S4->D3 S5->D3

  Flags      0540100002000240

    Types    Filter Enumerated ValidPnP

    Caps     PCIBus

    Props    HasAddress Enabled AcpiPower

Dump the namespace object. Use /s to display the subtree under this object and look for a _PRT method.

kd> !amli dns /s fffffa80052e3890

 

ACPI Name Space: \_SB.PCI0.PEX7 (fffffa80052e3890)

Device(PEX7)

| Integer(_ADR:Value=0x0000000000070000[458752])

| Integer(_STA:Value=0x000000000000000f[15])

| Method(_PRT:Flags=0x0,CodeBuff=fffffa80052e3aa9,Len=144) << A _PRT method exists for this object

 

Now, we have a swizzled IntPin value of 3, and a pointer to the _PRT method.  We can move on to the next step.

 

Step 4: Convert the pin number from PCI to ACPI numbering

 

The !pci or !devext output, and subsequent swizzling will show pin numbering in PCI format where 1 = INTA. But the ACPI table uses 0 for INTA. So you need to subtract one from the PCI pin number to get the ACPI pin number.

 

PCI pin number

ACPI pin number

INTA

1

0

INTB

2

1

INTC

3

2

INTD

4

3

 

Once you’ve converted to ACPI pin numbering, you have to dump the _PRT method to find the package which maps to that pin number.

 

For my example since the PCI IntPin value is 3, which corresponds to INTC, the ACPI pin number is 2

 

Step 5: Parse the _PRT method to find the static routing table

Now that we located the correct _PRT entry, we need to use the AMLI debugger extension to parse the method and find the static routing table.  The command !amli u will unassemble an ACPI method

kd> !amli u \_SB.PCI0.PEX7._PRT

AMLI_DBGERR: Failed to get address of ACPI!gDebugger

 

fffffa80052e3aa9 : If(LNot(PICF))

fffffa80052e3ab1 : {

fffffa80052e3ab1 : | Name(P10B, Package(0x4)

fffffa80052e3ab9 : | {

fffffa80052e3ab9 : | | Package(0x4)

fffffa80052e3abc : | | {

fffffa80052e3abc : | | | 0xffff,

fffffa80052e3abf : | | | 0x0,

fffffa80052e3ac1 : | | | LK00,

fffffa80052e3ac5 : | | | 0x0

fffffa80052e3ac7 : | | },

fffffa80052e3ac7 : | | Package(0x4)

fffffa80052e3aca : | | {

fffffa80052e3aca : | | | 0xffff,

fffffa80052e3acd : | | | 0x1,

fffffa80052e3acf : | | | LK01,

fffffa80052e3ad3 : | | | 0x0

fffffa80052e3ad5 : | | },

fffffa80052e3ad5 : | | Package(0x4)

fffffa80052e3ad8 : | | {

fffffa80052e3ad8 : | | | 0xffff,

fffffa80052e3adb : | | | 0x2,

fffffa80052e3add : | | | LK02,

fffffa80052e3ae1 : | | | 0x0

fffffa80052e3ae3 : | | },

fffffa80052e3ae3 : | | Package(0x4)

fffffa80052e3ae6 : | | {

fffffa80052e3ae6 : | | | 0xffff,

fffffa80052e3ae9 : | | | 0x3,

fffffa80052e3aeb : | | | LK03,

fffffa80052e3aef : | | | 0x0

fffffa80052e3af1 : | | }

fffffa80052e3af1 : | })

fffffa80052e3af1 : | Store(P10B, Local0)

fffffa80052e3af7 : }

fffffa80052e3af7 : Else

fffffa80052e3af9 : {

fffffa80052e3af9 : | Name(A10B, Package(0x4)

fffffa80052e3b01 : | {

fffffa80052e3b01 : | | Package(0x4)

fffffa80052e3b04 : | | {

fffffa80052e3b04 : | | | 0xffff,

fffffa80052e3b07 : | | | 0x0,

fffffa80052e3b09 : | | | 0x0,

fffffa80052e3b0b : | | | 0x26

fffffa80052e3b0d : | | },

fffffa80052e3b0d : | | Package(0x4)

fffffa80052e3b10 : | | {

fffffa80052e3b10 : | | | 0xffff,

fffffa80052e3b13 : | | | 0x1,

fffffa80052e3b15 : | | | 0x0,

fffffa80052e3b17 : | | | 0x2d

fffffa80052e3b19 : | | },

fffffa80052e3b19 : | | Package(0x4)

fffffa80052e3b1c : | | {

fffffa80052e3b1c : | | | 0xffff,

fffffa80052e3b1f : | | | 0x2,

fffffa80052e3b21 : | | | 0x0,

fffffa80052e3b23 : | | | 0x2f

fffffa80052e3b25 : | | },

fffffa80052e3b25 : | | Package(0x4)

fffffa80052e3b28 : | | {

fffffa80052e3b28 : | | | 0xffff,

fffffa80052e3b2b : | | | 0x3,

fffffa80052e3b2d : | | | 0x0,

fffffa80052e3b2f : | | | 0x2e

fffffa80052e3b31 : | | }

fffffa80052e3b31 : | })

fffffa80052e3b31 : | Store(A10B, Local0)

fffffa80052e3b37 : }

fffffa80052e3b37 : Return(Local0)

fffffa80052e3b39 : Zero

fffffa80052e3b3a : Zero

fffffa80052e3b3b : Zero

fffffa80052e3b3c : Zero

fffffa80052e3b3d : Zero

fffffa80052e3b3e : Zero

fffffa80052e3b3f : Zero

fffffa80052e3b40 : HNSO

fffffa80052e3b44 : Not(Zero, )

fffffa80052e3b47 : Zero

fffffa80052e3b48 : Zero

fffffa80052e3b49 : AMLI_DBGERR: UnAsmOpcode: invalid opcode class 0

AMLI_DBGERR: Failed to unassemble scope at 382d4a0 (size=4096)

 

There are 2 different _PRT tables here, each with 4 packages (think of it as 2 arrays, each containing 4 structures). The first is using link nodes, the second is using static interrupts.  The first list is used if we are in PIC mode, the second if we are in APIC mode.

 

We can check the value of PICF to determine the mode. (I expect it to be APIC but let’s check)

kd> !amli dns \PICF

 

ACPI Name Space: \PICF (fffffa80052ded18)

Integer(PICF:Value=0x0000000000000001[1])

 

So we’re in APIC mode (PICF != 0), we use the static routing mode. So we will use the 2nd table.  What does each package represent?  Each is a PCI Routing Table. From ACPI spec section 6.2.12, which describes the _PRT:

Table 6-14   Mapping Fields

Field

Type

Description

Address

DWORD

The address of the device (uses the same format as _ADR).

Pin

BYTE

The PCI pin number of the device (0–INTA, 1–INTB, 2–INTC, 3–INTD).

Source

NamePath

Or

BYTE

Name of the device that allocates the interrupt to which the above pin is connected. The name can be a fully qualified path, a relative path, or a simple name segment that utilizes the namespace search rules. Note: This field is a NamePath and not a String literal, meaning that it should not be surrounded by quotes. If this field is the integer constant Zero (or a BYTE value of 0), then the interrupt is allocated from the global interrupt pool.

Source Index

DWORD

Index that indicates which resource descriptor in the resource template of the device pointed to in the Source field this interrupt is allocated from. If the Source field is the BYTE value zero, then this field is the global system interrupt number to which the pin is connected.

 

fffffa80052e3af9 : | Name(A10B, Package(0x4)

fffffa80052e3b01 : | {

fffffa80052e3b01 : | | Package(0x4)

fffffa80052e3b04 : | | {

fffffa80052e3b04 : | | | 0xffff,

fffffa80052e3b07 : | | | 0x0,          << INTA

fffffa80052e3b09 : | | | 0x0,

fffffa80052e3b0b : | | | 0x26  << Interrupt line

fffffa80052e3b0d : | | },

fffffa80052e3b0d : | | Package(0x4)

fffffa80052e3b10 : | | {

fffffa80052e3b10 : | | | 0xffff,

fffffa80052e3b13 : | | | 0x1,          << INTB

fffffa80052e3b15 : | | | 0x0,

fffffa80052e3b17 : | | | 0x2d

fffffa80052e3b19 : | | },

fffffa80052e3b19 : | | Package(0x4)

fffffa80052e3b1c : | | {

fffffa80052e3b1c : | | | 0xffff,

fffffa80052e3b1f : | | | 0x2,          << INTC

fffffa80052e3b21 : | | | 0x0,

fffffa80052e3b23 : | | | 0x2f

fffffa80052e3b25 : | | },

fffffa80052e3b25 : | | Package(0x4)

fffffa80052e3b28 : | | {

fffffa80052e3b28 : | | | 0xffff,

fffffa80052e3b2b : | | | 0x3,          << INTD

fffffa80052e3b2d : | | | 0x0,

fffffa80052e3b2f : | | | 0x2e

fffffa80052e3b31 : | | }

fffffa80052e3b31 : | })

 

Step 6 - Find the routing entry which represents our IntPin

 

Now, we just have to locate the entry in the routing table with a IntPin value of 2.

 

fffffa80052e3b19 : | | Package(0x4)

fffffa80052e3b1c : | | {

fffffa80052e3b1c : | | | 0xffff,

fffffa80052e3b1f : | | | 0x2, <<< IntPin 2 (INTC)

fffffa80052e3b21 : | | | 0x0,

fffffa80052e3b23 : | | | 0x2f << IRQ is 0x2f

 

So the device should be assigned IRQ 0x2F.   However, you may have noticed from the !pci output above that in this case the device was actually assigned IntLine (IRQ) 0x2e!  Since the wrong interrupt line was assigned after the device changed slots in the system, the device did not receive interrupts and hence was not functional.

 

I hope this was useful to help understand how interrupts are assigned to LBI devices.

 

More reading / references:

 

PCI IRQ Routing on a Multiprocessor ACPI System:

https://msdn.microsoft.com/en-us/windows/hardware/gg454523.aspx#EDD

 

ACPI 4.0 spec

https://www.acpi.info/DOWNLOADS/ACPIspec40a.doc

Comments

  • Anonymous
    September 01, 2011
    Nice writeup!. who assign the wrong int. 2e to that device? [ACPI.SYS did.  We have plans to correct this in the future.  We got the cart before the horse when we published this article before a solution was available.]

  • Anonymous
    September 07, 2011
    The comment has been removed

  • Anonymous
    September 20, 2011
    How the problem has been solved in this case?

  • Anonymous
    September 22, 2011
    Why ACPI.SYS changing that as it should only read tables from BIOS? Can you elaborate in what situation ACPI.sys will map to other intr.? [Typically we provide this information in the KB article that documents the fix.  I would rather not comment on the exact cause until the solution is finalized and published.  As I said earlier, we got the cart before the horse on this one; although the purpose of the article was to show how PCIe addressing works and not necessarily provide a deep dive on this problem. I'll update the article when the fix is available.]

  • Anonymous
    September 25, 2011
    Not a Issue. By the way Great analysis. Thanks Graham. Really good step by step discription.