Rabbit Bug Reports Questions? Problems? send mail to dheller@scl.ameslab.gov ----------------------------------------------------------------------------- Counter Bits Permuted Disabled Cache May Cause Crash Type Conversions May Be Expensive Dual Processor May Move Process Modinfo Does Not Find Loaded Module pmc_dev.c May Fail to Compile ----------------------------------------------------------------------------- Counter Bits Permuted There are three instructions involved with reading the cycle and event counters: rdmsr (read model-specific register), rdtsc (read time-stamp counter), and rdpmc (read performance-monitoring counter, available since the Pentium with MMX). rdmsr can only be executed in kernel mode; the /dev/msr module by Stephan Meyer is a convenient way to execute the instruction from a user program. rdtsc and rdpmc can be executed from user mode if the kernel grants permission through two control bits. In an earlier version of the rabbit program, /dev/msr was used to read both the cycle and event counters. On some occasions, often but not always associated with the interrupt handler used to capture control-C, the counter bits were permuted by the time they arrived in memory. It is not clear how this happened. When the time-stamp counter was read with rdtsc, and the event counters with /dev/msr, the TSC would arrive correctly in all cases, even when the PMC's were wrong. Since we started using rdtsc and rdpmc exclusively, these problems have not been observed. Here are some examples of output from the old program; please notify us if you encounter a similar problem. This is from a 200 MHz Pentium Pro. The output is edited to delete extraneous material. Normal g% rabbit -g 0 ... ... Sample rate = 100 per second, samples taken = 340 Event Events Events/sec -------------------------- ---------------- ---------------- fpu_op_executed 200017242 59348499.38 data_mem_refs 233586793 69309153.05 Abnormal, same program g% rabbit -g 0 ... ... Sample rate = 100 per second, samples taken = 341 Event Events Events/sec -------------------------- ---------------- ---------------- fpu_op_executed 650552426333 9.19 data_mem_refs 320050333134 4.52 Abnormal, another program with -DPMC_LIST compile-time option TSC is read with rdtsc, PMC's with rdmsr g% rabbit -e 194,194 -s 1 ... ... read 0 00004f:5db126a0ba [TSC] read 1 00004f:99291b366a [PMC 0] read 2 00004f:0d6499a564 [PMC 1] read 0 00004f:5dbc8a9667 read 1 00004f:9940c9519d read 2 00004f:0d7c479e29 subtract 0 000000:000b63f5ad [t1 - t0, 0.95550 sec.] subtract 1 000000:0017ae1b33 subtract 2 000000:0017adf8c5 read 0 00004f:5dbc8cdddf read 1 00004f:9940cc730b read 2 00004f:0d7c4abf5b read 0 00004f:5dbd303174 [bits ok] read 1 40c951:9d00004f99 [bits permuted] read 2 7c479e:2900004f0d [bits permuted] subtract 0 000000:0000a35395 [t1 - t0, 0.05351 sec.] subtract 1 000000:03bf33dc8e subtract 2 000000:1b83b58fb2 read 0 00004f:5dbd324d67 [bits ok] read 1 40cc73:0b00004f99 [bits permuted] read 2 7c4abf:5b00004f0d [bits permuted] Sample rate = 1 per second, samples taken = 2 Event Events Events/sec -------------------------- ---------------- ---------------- uops_retired 16490035137 16342616726.25 uops_retired 118571108471 117511100761.69 ----------------------------------------------------------------------------- Disabled Cache May Cause Crash The pmc_configure() routine allows various privileged actions, some of which can have unfortunate consequences. If the cache is disabled by pmc_configure(PMC_CONFIGURE_CACHE, PMC_CONFIGURE_OFF); and later enabled by pmc_configure(PMC_CONFIGURE_CACHE, PMC_CONFIGURE_ON); we might hope to discover something about the system design. However, there is probably some timing operation in the OS or a device driver that will run slowly, and this could cause a fatal error, as has been observed. The pmc_open(), and pmc_close() functions will enable the cache, so an execution of rabbit or pmc_query will return the system to a safe state. Nevertheless, we recommend that the cache not be disabled on any system where you do not have direct physical access to the power switch. If this feature represents a security threat to your system, then read the /dev/pmc code in pmc_dev.c, and the supporting code in pmc_lib.c, and remove anything you do not trust. The easy way to do this is to not select the compile-time option -DPMC_ALLOW_CACHE_DISABLE in the Makefile. ----------------------------------------------------------------------------- Type Conversions May Be Expensive Since a cycle counter is inherently unsigned, the declaration in pmc_lib.h is typedef unsigned long long int pmc_cycle_t; To convert cycles to seconds, we use double pmc_seconds_per_cycle = 0.000000005; /* 200 MHz */ double pmc_second(const pmc_cycle_t c) { return (double)(c) * pmc_seconds_per_cycle; } The code generated by this little routine is surprisingly long, since there is no x86 instruction to convert unsigned integers to floating point, and the fild instructions (which convert signed integers to floating point) only take memory operands. .globl __cmpdi2 .section .rodata .align 4 .LC0: .long 0x0,0x80000000,0x403f .text .align 4 .globl pmc_second .type pmc_second,@function pmc_second: pushl %ebp movl %esp,%ebp subl $20,%esp pushl %esi pushl %ebx movl 8(%ebp),%edx movl 12(%ebp),%ecx pushl %ecx pushl %edx fildll (%esp) addl $8,%esp fstpt -12(%ebp) pushl $0 pushl $0 pushl %ecx pushl %edx call __cmpdi2 cmpl $1,%eax jge .L54 fldt .LC0 fldt -12(%ebp) faddp %st,%st(1) fstpt -12(%ebp) .L54: fldt -12(%ebp) subl $8,%esp fstpl (%esp) popl %ebx popl %esi movl %ebx,-20(%ebp) movl %esi,-16(%ebp) fldl -20(%ebp) fmull pmc_seconds_per_cycle leal -28(%ebp),%esp popl %ebx popl %esi leave ret If we were willing to abandon one bit of the cycle counter, the code would become typedef long long int pmc_cycle_t; .align 4 .globl pmc_second .type pmc_second,@function pmc_second: pushl %ebp movl %esp,%ebp fildll 8(%ebp) fmull pmc_seconds_per_cycle leave ret But, of course, something else would become more complicated, or break. This may be a good compromise: double pmc_second(const pmc_cycle_t c) { double e = (long long int) c; if (e < 0) { e = 0x8000000000000000LL - e; } return (e * pmc_seconds_per_cycle); } which compiles to .section .rodata .align 4 .LC1: .long 0x0,0x43e00000 .text .align 4 .globl pmc_second .type pmc_second,@function pmc_second: pushl %ebp movl %esp,%ebp fildll 8(%ebp) fldz fcomp %st(1) fnstsw %ax andb $69,%ah jne .L56 fldl .LC1 fsubp %st,%st(1) .L56: fmull pmc_seconds_per_cycle leave ret ----------------------------------------------------------------------------- Dual Processor May Move Process The SMP version of rabbit has not been completed. You can expect some problems since the measurements are taken on a processor, but a process is not locked to a processor. There are kernel modifications available which get around this "feature" by putting the performance counters into the process state. Another approach (which I prefer) is to allow a process to lock itself and all its children onto its current processor. If you like reading kernel source code, look at schedule() in kernel/sched.c, and do_fork() in kernel/fork.c, paying attention to the has_cpu, processor and last_processor fields of task_struct, which is defined in include/linux/sched.h. Here is an experiment to try on a dual processor system: rabbit -v > 0 & rabbit -v > 1 & You should find information on each processor in the output files, similar to this: % more 0 1 :::::::::::::: 0 :::::::::::::: processor information: GenuineIntel Pentium-class processor Dual processor, P5 family, Pentium/MMX (P55C), stepping 3 Pentium/MMX (the above information may not be definitive) rdtsc instruction from any CPL rdpmc instruction from any CPL Indicated features 0 Floating Point Unit 1 Virtual-8086 Mode Extension 2 Debugging Extension 3 Page Size Extension 4 Time Stamp Counter 5 Model Specific Registers 7 Machine Check Exception 8 CMPXCHG8B Instruction 9 Local APIC Hardware Enabled 23 MMX rabbit: Performance-Monitoring Counters Library, for Intel Processors and Linux Pentium/MMX version 1.00, 25 February 1999, Ames Laboratory, US DOE :::::::::::::: 1 :::::::::::::: processor information: GenuineIntel Pentium-class processor Original OEM processor, P5 family, Pentium/MMX (P55C), stepping 3 Pentium/MMX (the above information may not be definitive) rdtsc instruction from any CPL rdpmc instruction from any CPL Indicated features 0 Floating Point Unit 1 Virtual-8086 Mode Extension 2 Debugging Extension 3 Page Size Extension 4 Time Stamp Counter 5 Model Specific Registers 7 Machine Check Exception 8 CMPXCHG8B Instruction 9 Local APIC Hardware Enabled 23 MMX rabbit: Performance-Monitoring Counters Library, for Intel Processors and Linux Pentium/MMX version 1.00, 25 February 1999, Ames Laboratory, US DOE ----------------------------------------------------------------------------- Modinfo Does Not Find Loaded Module The modinfo command is supposed to supply data from the MODULE_AUTHOR() and MODULE_DESCRIPTION() macros. It doesn't find the module, even when lsmod will. % lsmod pmc_dev 4808 0 lockd 30344 1 (autoclean) sunrpc 52132 1 (autoclean) [lockd] 3c59x 18980 1 (autoclean) es1371 25092 0 soundcore 2628 4 [es1371] % cat /proc/misc 200 /dev/pmc 135 rtc 134 apm 1 psaux % cat /proc/pmc /dev/pmc owner pid: (none) % modinfo -a lockd Olaf Kirch % modinfo -a pmc_dev modinfo: pmc_dev: no module by that name found However, 'modinfo -a /usr/local/lib/pmc_dev.o' does the right thing, as does % objdump --full-contents --section .modinfo pmc_dev.o pmc_dev.o: file format elf32-i386 Contents of section .modinfo: 0000 6b65726e 656c5f76 65727369 6f6e3d32 kernel_version=2 0010 2e322e31 342d352e 30007573 696e675f .2.14-5.0.using_ 0020 63686563 6b73756d 733d3100 00000000 checksums=1..... 0030 00000000 00000000 00000000 00000000 ................ 0040 61757468 6f723d44 6f6e2048 656c6c65 author=Don Helle 0050 72203c64 68656c6c 65724073 636c2e61 r ..... 0070 00000000 00000000 00000000 00000000 ................ 0080 64657363 72697074 696f6e3d 496e7465 description=Inte 0090 6c207065 72666f72 6d616e63 6520636f l performance co 00a0 756e7465 72732c20 61636365 73732061 unters, access a 00b0 6e642063 6f6e7472 6f6c00 nd control. ----------------------------------------------------------------------------- pmc_dev.c May Fail to Compile Some Linux distributions, in combination with local installations of the gcc and egcs compilers, may not find certain system include files, or the files may not provide certain typedefs. In particular, RedHat has been shipping the GCC 2.96 compiler, which is experimental (see http://www.gnu.org/software/gcc/gcc-2.96.html dated October 6th, 2000). The GCC versions egcs-2.91.66 and 2.95.3 work just fine with the most recent release of Rabbit. ----------------------------------------------------------------------------- Last revised 22 May 2001