Programming Manual
Page 2
... will not operate (including 32-bit operation) without notice. Copies of Intel Corporation or its subsidiaries in developer's software code when running on the specific hardware and software you use. Intel may be claimed as errata. Intel® Virtualization Technology requires a computer system with an Intel® processor supporting Hyper-Threading Technology and an HT Technology enabled chipset, BIOS and operating system. NO...
... will not operate (including 32-bit operation) without notice. Copies of Intel Corporation or its subsidiaries in developer's software code when running on the specific hardware and software you use. Intel may be claimed as errata. Intel® Virtualization Technology requires a computer system with an Intel® processor supporting Hyper-Threading Technology and an HT Technology enabled chipset, BIOS and operating system. NO...
Programming Manual
Page 14
... is identified, the last step is to read that software freeze and disable the counters (by setting the U_MSR_PMON_GLOBAL_CTL.en_all bit to read the S{0,1}_MSR_PMON_SUMMARY.ov_* bits. i.e. If polling, skip to gather data. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE d) Enable counting at the box-level: Enable counters within the Intel Xeon Processor 7500 Series uncore. set U_MSR_PMON_GLOBAL_CTL.en_all to 0 to...
... is identified, the last step is to read that software freeze and disable the counters (by setting the U_MSR_PMON_GLOBAL_CTL.en_all bit to read the S{0,1}_MSR_PMON_SUMMARY.ov_* bits. i.e. If polling, skip to gather data. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE d) Enable counting at the box-level: Enable counters within the Intel Xeon Processor 7500 Series uncore. set U_MSR_PMON_GLOBAL_CTL.en_all to 0 to...
Programming Manual
Page 18
... by the U-Box, as they arbitrate to B-Box Active Request Cycles 1 Number of core woken up 1 Number of times cores were sent IPIs or were Woken up. - U_MSR_PMON_CTR Register - INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING The U-Box performance monitor data register is detected. N and setting the control register to send a PMI to the Ring or a B-Box. Errors detected and distinguished...
... by the U-Box, as they arbitrate to B-Box Active Request Cycles 1 Number of core woken up 1 Number of times cores were sent IPIs or were Woken up. - U_MSR_PMON_CTR Register - INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING The U-Box performance monitor data register is detected. N and setting the control register to send a PMI to the Ring or a B-Box. Errors detected and distinguished...
Programming Manual
Page 20
.../Cyc: 1, • Definition: Number of core IPIs sent. UNCORRECTED_ERR • Title: Uncorrected Error • Category: U-Box Events • Event Code: 0x1E5, Max. UNCORE PERFORMANCE MONITORING 2-8 INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE FATAL_ERR • Title: Fatal Errors • Category: U-Box Events • Event Code: 0x1E6, Max. Inc/Cyc: 1, • Definition: Number of cores woken up. Inc/Cyc: 1, • Definition: Number of uncorrected errors. U2B_REQUEST_CYCLES • Title...
.../Cyc: 1, • Definition: Number of core IPIs sent. UNCORRECTED_ERR • Title: Uncorrected Error • Category: U-Box Events • Event Code: 0x1E5, Max. UNCORE PERFORMANCE MONITORING 2-8 INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE FATAL_ERR • Title: Fatal Errors • Category: U-Box Events • Event Code: 0x1E6, Max. Inc/Cyc: 1, • Definition: Number of cores woken up. Inc/Cyc: 1, • Definition: Number of uncorrected errors. U2B_REQUEST_CYCLES • Title...
Programming Manual
Page 21
... the LLC are directed from a C-Box performance counter, the overflow bit is detected. This enables the individual CBox instances to the U-Box when an overflow is set associative slice of the C-Box For the Intel Xeon Processor 7500 Series, the LLC coherence engine (C-Box) manages the interface between the cores within the socket that pass through the socket's LLC remain coherent. Refer to...
... the LLC are directed from a C-Box performance counter, the overflow bit is detected. This enables the individual CBox instances to the U-Box when an overflow is set associative slice of the C-Box For the Intel Xeon Processor 7500 Series, the LLC coherence engine (C-Box) manages the interface between the cores within the socket that pass through the socket's LLC remain coherent. Refer to...
Programming Manual
Page 28
... of core caches Internal C-Box Queues: IRQ - Associated with snoops from C to C-Box. Ingress Data Queue on AD Ring. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING 2.3.4 C-BOX Performance Monitoring Events 2.3.4.1 An Overview: The performance monitoring events within the C-Box include all events internal to the LLC as well as events which track ring related activity at the C-Box/Core sink inject...
... of core caches Internal C-Box Queues: IRQ - Associated with snoops from C to C-Box. Ingress Data Queue on AD Ring. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING 2.3.4 C-BOX Performance Monitoring Events 2.3.4.1 An Overview: The performance monitoring events within the C-Box include all events internal to the LLC as well as events which track ring related activity at the C-Box/Core sink inject...
Programming Manual
Page 29
... Max Code Inc/Cyc Description Ring Events BOUNCES_P2C_AD BOUNCES_C2P_AK BOUNCES_C2P_BL 0x01 0x02 0x03 1 Number of P2C AD bounces. 1 Number of C2P AK bounces. 1 Number of these buffers informs how many transactions the C-Box is 5bits wide and dedicated to -one -to its queue: IRQ, IPQ, VIQ, MAF, RWRF, RSPF, IDF. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE...
... Max Code Inc/Cyc Description Ring Events BOUNCES_P2C_AD BOUNCES_C2P_AK BOUNCES_C2P_BL 0x01 0x02 0x03 1 Number of P2C AD bounces. 1 Number of C2P AK bounces. 1 Number of these buffers informs how many transactions the C-Box is 5bits wide and dedicated to -one -to its queue: IRQ, IPQ, VIQ, MAF, RWRF, RSPF, IDF. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE...
Programming Manual
Page 31
INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING ARB_LOSSES • Title: Arbiter Losses. • Category: Ring - b00010000 BL ring in the direction that points toward the nearest S-Box b00000010 AD ring in the direction that points away from the nearest S-Box...Event Code: 0x0A, Max. Inc/Cyc: 7, • Definition: Number of Ring arbitration losses. Egress • Event Code: 0x09, Max. Inc/Cyc: 7, • Definition: Number of Ring arbitration wins. b00010000 BL ring in the direction that points toward the nearest S-Box ...
INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING ARB_LOSSES • Title: Arbiter Losses. • Category: Ring - b00010000 BL ring in the direction that points toward the nearest S-Box b00000010 AD ring in the direction that points away from the nearest S-Box...Event Code: 0x0A, Max. Inc/Cyc: 7, • Definition: Number of Ring arbitration losses. Egress • Event Code: 0x09, Max. Inc/Cyc: 7, • Definition: Number of Ring arbitration wins. b00010000 BL ring in the direction that points toward the nearest S-Box ...
Programming Manual
Page 32
...*) b000000x1 Direction that points toward the nearest S-Box b0000001x Direction that bounced on the BL ring. Inc/Cyc: 1, • Definition: Number of a processor's cache that points away from the nearest S-Box b00000011 Either direction 2-20 Extension --SB NSB... Event Code: 0x03, Max. WIR • Event Code: 0x04, Max. WIR • Event Code: 0x01, Max. Inc/Cyc: 1, • Definition: Number of C-Box snoops of LLC data responses to the core that bounced on AD ring. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING ...
...*) b000000x1 Direction that points toward the nearest S-Box b0000001x Direction that bounced on the BL ring. Inc/Cyc: 1, • Definition: Number of a processor's cache that points away from the nearest S-Box b00000011 Either direction 2-20 Extension --SB NSB... Event Code: 0x03, Max. WIR • Event Code: 0x04, Max. WIR • Event Code: 0x01, Max. Inc/Cyc: 1, • Definition: Number of C-Box snoops of LLC data responses to the core that bounced on AD ring. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING ...
Programming Manual
Page 34
... (*nothing will be counted*) b00000xx1 Shared - Inc/Cyc: 1, • Definition: Last Level Cache Misses Extension --S F I umask [15:8] Description b00000000 (*nothing will be upgraded (due to RFO) b000001xx Invalid - LLC • Event Code: 0x16, Max. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING Extension F ALL umask [15:8] Description b00001xxx Forward (S with right to Forward on...
... (*nothing will be counted*) b00000xx1 Shared - Inc/Cyc: 1, • Definition: Last Level Cache Misses Extension --S F I umask [15:8] Description b00000000 (*nothing will be upgraded (due to RFO) b000001xx Invalid - LLC • Event Code: 0x16, Max. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING Extension F ALL umask [15:8] Description b00001xxx Forward (S with right to Forward on...
Programming Manual
Page 35
... was delayed because the transaction had a snoop pending. b11111111 Total number of them gets delayed. MAF • Event Code: 0x12, Max. MAF • Event Code: 0x11, Max. Extension --GO_PENDING umask [15:8] Description b00000000 (*nothing will be counted*) bxxxxxxx1 An incoming local processor RD/WR or remote Intel QPI snoop request that MAF_NACK1 with a transaction monitored by the...
... was delayed because the transaction had a snoop pending. b11111111 Total number of them gets delayed. MAF • Event Code: 0x12, Max. MAF • Event Code: 0x11, Max. Extension --GO_PENDING umask [15:8] Description b00000000 (*nothing will be counted*) bxxxxxxx1 An incoming local processor RD/WR or remote Intel QPI snoop request that MAF_NACK1 with a transaction monitored by the...
Programming Manual
Page 37
... the transfer of messages sunk from the core to processor) SINKS_P2C • Title: P2C Sinks • Category: Ring - INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING OCCUPANCY_RWRF • Title: RWRF Occupancy • Category: Queue Occupancy • Event Code: 0x20, Max. Inc/Cyc: 3, • Definition: Number of 32 bytes, or 2 sinks per cache line. Inc/Cyc: 8, • Definition: Cumulative...
... the transfer of messages sunk from the core to processor) SINKS_P2C • Title: P2C Sinks • Category: Ring - INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING OCCUPANCY_RWRF • Title: RWRF Occupancy • Category: Queue Occupancy • Event Code: 0x20, Max. Inc/Cyc: 3, • Definition: Number of 32 bytes, or 2 sinks per cache line. Inc/Cyc: 8, • Definition: Cumulative...
Programming Manual
Page 38
... ring at the C-Box that were sent by the processor that hit in remote cache. WIR • Event Code: 0x07, Max. GotoI: LLC Data Read for Ownership Snoop Hit 'x' state in the LLC according to RD requests) SINKS_S2P_BL • Title: S2P Sinks • Category: Ring - This cov- INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING SINKS_S2C •...
... ring at the C-Box that were sent by the processor that hit in remote cache. WIR • Event Code: 0x07, Max. GotoI: LLC Data Read for Ownership Snoop Hit 'x' state in the LLC according to RD requests) SINKS_S2P_BL • Title: S2P Sinks • Category: Ring - This cov- INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING SINKS_S2C •...
Programming Manual
Page 40
... Queue. All LLC victims pass through the Snoop Response FIFO. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING TRANS_IRQ • Title: IRQ Transactions • Category: Queue Occupancy • Event Code: 0x19, Max. Inc/Cyc: 1, • Definition: Number of snoop responses from the processor that there is back pressure due to allocate entries in the...
... Queue. All LLC victims pass through the Snoop Response FIFO. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING TRANS_IRQ • Title: IRQ Transactions • Category: Queue Occupancy • Event Code: 0x19, Max. Inc/Cyc: 1, • Definition: Number of snoop responses from the processor that there is back pressure due to allocate entries in the...
Programming Manual
Page 41
... a specific set of events as specified in the Intel® QuickPath Interconnect Specification). Each of these four counters is dedicated to monitor events) and the overflow bit(s) has been cleared, the B-Box is prepared for a new sample interval. HW can be also configured (by a maximum of 1 per cycle. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING 2.4 B-Box Performance Monitoring...
... a specific set of events as specified in the Intel® QuickPath Interconnect Specification). Each of these four counters is dedicated to monitor events) and the overflow bit(s) has been cleared, the B-Box is prepared for a new sample interval. HW can be also configured (by a maximum of 1 per cycle. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING 2.4 B-Box Performance Monitoring...
Programming Manual
Page 57
... a new freeze, the overflow field responsible for converting C-box requests to ring) connections. Assuming all uncore counting and/or send a PMI to Section 2.1, "Global Performance Monitoring Control". 2.5.2.1 S-Box PMU - Each S-Box collects overflow bits for all boxes on how to setup a monitoring session, refer to selected cores when it will resume. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING 2.5 S-Box Performance...
... a new freeze, the overflow field responsible for converting C-box requests to ring) connections. Assuming all uncore counting and/or send a PMI to Section 2.1, "Global Performance Monitoring Control". 2.5.2.1 S-Box PMU - Each S-Box collects overflow bits for all boxes on how to setup a monitoring session, refer to selected cores when it will resume. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING 2.5 S-Box Performance...
Programming Manual
Page 85
... 2.6.2 R-Box Performance Monitoring Overview The R-Box supports performance event monitoring through Machine Specific Registers (MSRs). R-Box PMUs are split. At a high level, the R-Box PMU supports features comparable to other peripheral control registers. R-Box PMUs support both Global and Local PMU freeze/unfreeze. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING 2.6.1.3 R-Box Output Port The R-Box output port acts as a virtual wire that allow a user...
... 2.6.2 R-Box Performance Monitoring Overview The R-Box supports performance event monitoring through Machine Specific Registers (MSRs). R-Box PMUs are split. At a high level, the R-Box PMU supports features comparable to other peripheral control registers. R-Box PMUs support both Global and Local PMU freeze/unfreeze. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING 2.6.1.3 R-Box Output Port The R-Box output port acts as a virtual wire that allow a user...
Programming Manual
Page 100
...1 Starvation Detective 2.6.6 R-Box Performance Monitor Event List This section enumerates Intel Xeon Processor 7500 Series uncore performance monitoring events for R-Box Events Symbol Name Event Code Max Inc/Cyc Description RIX Events...New Packets Received by Port 1 Input Queue Read Win 1 Transactions allocated to ARB • Category: RIX • [Bit(s)] Value: See Note, Max. ALLOC_TO_ARB • Title: Transactions allocated to Arb 1 Cycles EOT Not Empty 1 EOT Occupancy 1 Number of Message Class [15:9] may be monitored. 2-88 Performance Monitor Events for the R-Box...
...1 Starvation Detective 2.6.6 R-Box Performance Monitor Event List This section enumerates Intel Xeon Processor 7500 Series uncore performance monitoring events for R-Box Events Symbol Name Event Code Max Inc/Cyc Description RIX Events...New Packets Received by Port 1 Input Queue Read Win 1 Transactions allocated to ARB • Category: RIX • [Bit(s)] Value: See Note, Max. ALLOC_TO_ARB • Title: Transactions allocated to Arb 1 Cycles EOT Not Empty 1 EOT Occupancy 1 Number of Message Class [15:9] may be monitored. 2-88 Performance Monitor Events for the R-Box...
Programming Manual
Page 104
... 2-59. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING NEW_PACKETS_RECV • Title: New Packets Received by Port • Category: RIX • [Bit(s)] Value: see table, Max. Inc/Cyc: 1, • Definition: Counts new packets received according to the Virtual Network and Message...b11xx1xxx b11x1xxxx b111xxxxx b11111111 Description VN0 Non-Coherent Bypass Messages VN0 Non-Coherent Standard Messages VN0 Data Response Messages VN0 Non-Data Response Messages VN0 Snoop Messages VN0 Home Messages VN0 All Messages VN1 Non-Coherent Bypass ...
... 2-59. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING NEW_PACKETS_RECV • Title: New Packets Received by Port • Category: RIX • [Bit(s)] Value: see table, Max. Inc/Cyc: 1, • Definition: Counts new packets received according to the Virtual Network and Message...b11xx1xxx b11x1xxxx b111xxxxx b11111111 Description VN0 Non-Coherent Bypass Messages VN0 Non-Coherent Standard Messages VN0 Data Response Messages VN0 Non-Data Response Messages VN0 Snoop Messages VN0 Home Messages VN0 All Messages VN1 Non-Coherent Bypass ...
Programming Manual
Page 133
... new scheduler commands that were accepted. NOTE: THR Bits [6:4] must be programmed with the DIMM # Advance the counter when the above mid temp thermal trip point (rising) is crossed in the "starved" state. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING RETRY_MFULL • Title: Retry MFull • Category: Retry Events • Event Code: 0x02, Max...
... new scheduler commands that were accepted. NOTE: THR Bits [6:4] must be programmed with the DIMM # Advance the counter when the above mid temp thermal trip point (rising) is crossed in the "starved" state. INTEL® XEON® PROCESSOR 7500 SERIES UNCORE PROGRAMMING GUIDE UNCORE PERFORMANCE MONITORING RETRY_MFULL • Title: Retry MFull • Category: Retry Events • Event Code: 0x02, Max...