main

• | Table of Contents
• | Index
• | Reviews
• | Reader Reviews
• | Errata
System Performance Tuning, Second Edition
By Mike Loukides, Gian-Paolo D. Musumeci
Publisher | : O'Reilly
Pub Date | : February 2002
ISBN | : 0-596-00284-X
Pages | : 336
Slots | : 1

System Performance Tuning covers two distinct areas: performance tuning, or the art of increasing performance for a specific application, and capacity planning, or deciding what hardware best fulfills a given role. Underpinning both subjects is the science of computer architecture. This book focuses on the operating system, the underlying hardware, and their interactions. For system administrators who want a hands-on introduction to system performance, this is the book to recommend.


Free Open Study

Colophon

Our look is the result of reader comments, our own experimentation, and feedback from distribution channels. Distinctive covers complement our distinctive approach to technical topics, breathing personality and life into potentially dry subjects.

The animal on the cover of System Performance Tuning, Second Edition is a swordfish, a marine fish with a long, sword-shaped snout. The swordfish is distinguished from the marlin by its flattened snout, high dorsal fin, toothless mouth, and lack of scales. The fastest swimmer of all fish, a swordfish can grow up to 14 feet in length and weigh a thousand pounds or more. This combination of speed and size gives it strength and momentum -- enough to drive its sword through the planking of a boat. The swordfish hunts by charging through schools of fish, striking to either side with its sword and then returning to collect its kill.

Colleen Gorman was the production editor and the copyeditor for System Performance Tuning, Second Edition. Mary Brady was the proofreader, and Linley Dolby provided quality control. Tom Dinse wrote the index.

Edie Freedman designed the cover of this book. The cover image is a 19th-century engraving from the Dover Pictorial Archive. Emma Colby produced the cover layout with QuarkXPress 4.1 using Adobe's ITC Garamond font.

Melanie Wang designed the interior layout, based on a series design by David Futato. Mihaela Maier converted the files using tools created by Mike Sierra. The text font is Linotype Birka; the heading font is Adobe Myriad Condensed; and the code font is LucasFont's TheSans Mono Condensed. The illustrations that appear in the book were produced by Robert Romano and Jessamyn Read using Macromedia FreeHand 9 and Adobe Photoshop 6. The tip and warning icons were drawn by Christopher Bing. This colophon was written by Colleen Gorman.

The online edition of this book was created by the Safari production group (John Chodacki, Becki Maisch, and Madeleine Newell) using a set of Frame-to-XML conversion and cleanup tools written and maintained by Erik Ray, Benn Salter, John Chodacki, and Jeff Liggett.


Free Open Study

Copyright © 2002 O'Reilly & Associates, Inc. All rights reserved.

Printed in the United States of America.

Published by O'Reilly & Associates, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O'Reilly & Associates books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safari.oreilly.com). For more information contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

The O'Reilly logo is a registered trademark of O'Reilly & Associates, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps.

While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.


index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]


index_A

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

access patterns (disks)
acctcom command, reviewing process accounting records
ACK bit (TCP)
adb command, kernel variable tuning
address mask
addresses
frames (networks)
IP
duplicates
memory
address space layout
limitations
SCSI bus
world wide name (Fibre Channel)
algorithms
asymptotic complexity measure
big O notation
Boyer-Moore searching
CRC (cyclical redundancy checking)
elevator
inode cache (UFS)
Kesslers Best Bin
Knuth-Morris-Pratt searching
Linux, epochs
majority consensus (Solstice DiskSuite)
Nagles 2nd
naive
page coloring
problem solving and
selecting
slow start (TCP)
string searching
Alpha processors (DEC)
memory page size
ALU (arithmetic and logic unit)
application layer (OSI model)
applications [See also software]
64-bit architecture
algorithms, selecting
analyzing
compiling with profiling
executing with profiling
profiling
compiler flags
floating-point behavior
Fortran
function pointers
UltraSPARC-III applications
multiprocessor systems, design issues
performance
books about
MIPS
ptime command
time command
timex command
profiling considerations
refactored code
arbitrated loop Fibre Channel topology (FC-AL)
architecture
64-bit 2nd
books about
disks
FC (Fibre Channel)
filesystems
FireWire
hardware level
IDE
interfaces
IPI (Intelligent Peripheral Interface)
levels of transformation
MBus
memory address space layout
memory hierarchy and caching
multiprocessor systems
networks
OSI model
peripheral buses
processors
caching
clock rates
design basics
piplining
process scheduling
SCSI (Small Computer System Interface)
UFS (Unix Filesystem)
virtual memory
von Neumann model
XBus
arithmetic and logic unit (ALU)
arithmetic optimization
arrays (disks)
associative mapping
asymptotic complexity measure
asynchronous data transfers (SCSI)
Asynchronous Transfer Mode (ATM)
asynchronous writes (disks)
ATA (AT Attachment) [See IDE]
ATA Packet Interface (ATAPI)
ATA-2 interface
ATA-3 interface
ATAPI (ATA Packet Interface)
ATM (Asynchronous Transfer Mode)
atomic operations
attachment unit interface (AUI) cable
attenuation (fiberoptic cabling)
AUI (attachment unit interface) cable
auto-negotiation (Ethernet)
autodetection (RAID arrays)
automounter (NFS)
average memory utilization statistics
average seek time (disks)
average shared-memory space


index_B

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

bandwidth-delay product
barriers (threads)
base metric (SPECint_base95)
base time quantum
batch queuing programs
bdflush command, high disk service times and
benchmarking 2nd [See also testing]
books about
commercial workload
databases
decision support systems
disks
hdparm
iozone
multiple pass results
tiobench
e-commerce transactional web servers
Linpack benchmarks
memory
lmbench
STREAM
MFLOPS (megaflops)
MIPS
SPEC benchmarks
TCP (Transaction Processing Council) benchmarks
terminal driven database environments
user defined
automation considerations
problem set selection
rule creation
runtime selection
web performance
big O notation
bits (data paths)
blocked processes
books
application performance
benchmarking
computer architecture
Java code optimization
networking
routing
TCP
Unix
workload management
booting
processor failure and
RAID devices
bottlenecks
disks
blocked processes
locating
mutex locking
Boyer-Moore searching algorithms
branch delay slot (RISC processors)
bridges
broadcast frames
buffer cache
NFS
UFS
buffers
receive
send
burst speed
busses
CompactPCI
Fibre Channel connections
multiprocessor systems
PCI, architecture
peripheral interconnect
monitoring tools
SBus
architecture
burst transfer size
clock speed
interrupt processing
transfer mode
SCSI
addresses
command queuing
differential signaling
termination
transaction operations
utilization
busstat command, peripheral interconnect monitoring


index_C

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

cables
failures and error generation
IDE
cabling
10BASE2 Ethernet
10BASE5 Ethernet
AUI (attachment unit interface)
fiberoptic
UTP
CacheFS
cache parameters, changing
cache, removing
mount command options
NFS and
cachefsstat command, filesystem cache hit rates
caches
access times
associative mapping
cache-aligned block copy cache-busting
cache-busters
direct mapping
directories, creating
disks
enabling
DNLC (directory name lookup cache)
filesystem
bdflush command
cyclic caching
disk interaction constraints
fsflush command
memory consumption
priority paging
removing
hierarchy
L1
L2
lines
linked list cache-busting
locality
mapping
miss cost
multiprocessor systems
naming service, static performance issues
NFS
buffer
DNLC
inode
non-unit stride cache-busting
organization
parameters, changing
rnode (NFS)
size anomalies
UFS
buffer
disabling
inode
write-back behavior
write-through behavior
caching
capacity planning, defined
CDDI (Copper Distributed Data Interface)
cfsadmin command
cache directories, creating
options
characterization [See workflow characterization]
checksum (TCP headers)
chunks (RAID arrays)
Linux md package
CIDR (classless interdomain routing)
circuit switched protocols
circuits (ATM)
CISC (complex instruction set computing)
instruction set length
ckpacct command, process accounting
class A networks
class B networks
subnetting
class C networks
classes, IP addresses
classless interdomain routing (CIDR)
clients
NFS
bursty transfers
sequential transfers
TCP/IP
clock rates
critical path
processors
SBus
clusters, UFS (Unix Filesystem)
code [See programming code]
code morphing
collisions (Ethernet)
combined processor memory switch (CPMS)
command queuing (SCSI)
commercial workload benchmarks
communication
busses
crossbars
Fireplane
interconnectionless architectures
multiprocessor systems 2nd
CompactPCI
compilers
code optimization considerations
fast option
flags
floating-point behavior
Fortran applications
function pointers
UltraSPARC-III systems
profile feedback
xarch option
xchip option
xcrossfile option
xdepend option
xinlining option
xO option
xprefetch option
xsfpconst option
xvector option
compiling, profiling and
complex instruction set computing [See CISC]
computational grids
computer architecture [See architecture]
concentrator (FDDI network interface)
conditional branches
prediction
congestion window (TCP)
control units, von Neumann model
controller (SCSI)
Copper Distributed Data Interface (CDDI)
copy-on-write faults
CPI (cycles per instruction)
cpio command, RAID devices, copying root filesystem
CPMS (combined processor memory switch)
CPU [See processors]
cpustat command
cputrack command
CRC (cyclical redundancy checking) algorithm
critical path
critical sections
cron, performance analysis tool intervals and
crontab file
disk quotas, enabling
sar command, enabling
crossbars, multiprocessor systems
Crusoe processor (Transmeta) 2nd
cut-through switches
cycles per instruction [See CPI]
cyclic caching
cyclical redundancy checking (CRC) algorithm
cylinders


index_D

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

d-cache
DAC (dual-attachment concentrator)
data blocks (UFS)
data cache
data encapsulation (Ethernet)
data link (networks)
data link address
Data Link and Medium Access Control (DLMAC) group (Ethernet)
data link layer (OSI model)
data paths, bits
data transfer patterns (networks)
data transfer rates (TCP)
databases, benchmarks
DDR SDRAM modules
DEC Alpha processors
memory page size
decision support systems, benchmarking
defect management, disks
deferred acknowledgements (TCP)
demand paging
denial-of-service attacks, SYN flooding
design issues
applications, refactored code
NFS disk subsystems
processors
RAID arrays
subsystems
test design
development time to runtime ratio (code tuning)
differential signaling (SCSI)
diminishing returns (code optimization)
direct cache mapping
direct memory access (DMA)
Direct Rambus (RDRAM)
directory entry (UFS)
directory name lookup cache (DNLC) 2nd
disabling
disk arrays [See RAID]
disk quotas
disks
access patterns
architecture
arrays
asynchronous writes compared to synchronous
average seek time
benchmarking
hdparm
iozone
multiple pass results
tiobench
bottlenecks
blocked processes
locating
burst speed
caches
enabling
cluster size (UFS)
defect management
distributed parity-protected striping (RAID)
elevator algorithms
Fibre Channel
Hamming code arrays (RAID)
high I/O skew
high service time problems
internal transfer speed
compared to real speed
logical volumes
megabyte, size of
minfree parameter
mirrored striping (RAID)
mirroring
disk replacement
MTBF (Mean Time Between Failure)
MTTDI (Mean Time To Data Inaccessibility)
MTTDL (Mean Time To Data Loss)
MTTR (Mean Time To Repair)
NFS
paging and swapping problems
parity-protected striping (RAID)
parity-protected striping with independent disks (RAID)
partition statistics
performance monitoring tools, iostat
RAID levels
distributed parity-protected striping
Hamming code arrays
mirrored striping
mirroring
overview
parity-protected striping
parity-protected striping with independent disks
striping
random access and storage capacity issues
read/write latency
reads
rotational delay parameter
RPS misses
seek times, reducing
specifications
spindle synchronization 2nd
string utilization
striping (RAID)
web servers
workgroup servers and
write throttling
writes
ZBR (zone bit rate recording)
dispadmin command, process scheduling
dispatch queue, kernel threads
dispatch tables, tuning
distributed parity-protected striping
DMA (direct memory access)
DNLC (directory name lookup cache) 2nd
doubly indirect blocks (filesystems)
DRAM (dynamic random access memory)
dual-attachment concentrator (DAC)
duplex mode
dynamic performance analysis
dynamic priority
dynamic random access memory (DRAM)


index_E

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

e-commerce, benchmarking
E-series processors (Sun)
echo command, NFS threads
EDO (extended data output) memory
education
usage and performance agreements
users
EIDE (Enhanced IDE)
elvtune command, disk read/write latency
embedded controllers
environment
computational grids
importance of
terminal driven database, benchmarks
user limits on
epochs (Linux scheduling algorithm)
errors
kernel memory allocation
Message fragmentation needed
Ethernet
100BASE-T4
100BASE-TX
10BASE-T
5-4-3 rule
auto-negotiation
collisions
configuration parameters, changing
Gigabit
signal basics
status, displaying
topologies
executable data segment (memory)
executable text segment (memory)
execution, profiling and
ext2fs
ext3fs
extended data output (EDO) memory
extent-based filesystems


index_F

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

fabric (Fibre Channel topology)
fast instruction set computing (FISC)
fast-page mode (FPM) memory
FC (Fibre Channel), architecture
FC-AL (arbitrated loop Fibre Channel topology)
FDDI (Fiber Distributed Data Interface)
fiberoptic cabling
Fibre Channel (FC), architecture
file and metadata filesystem logging
file locking, NFS and
files
nsswitch.conf, static performance issues
remote swapfiles with NFS
resolve.conf, static performance issues
system, persisting maxpgio value changes
filesystems [See also NFS]
architecture
CacheFS
caches
hit rates, checking
removing
copying root to RAID device
DNLC (directory name lookup cache)
ext2fs
ext3fs
free space reservations
information about, determining
inodes
JFS (Journaled Filesystem)
logging
overview
Solaris
Solstice DiskSuite software
network, mounting
ReiserFS
tail packing
rotational delay parameter
seek times, reducing
Solaris, tmpfs
tunefs command
tuning, single-user workstations
VxFS
web servers
workgroup servers and
FIN_WAIT_2 timer (TCP)
find command (RAID devices), copying root filesystem
Fireplane (Sun)
FireWire, architecture
FISC (fast instruction set computing)
Fixed Shroud Device (FSD) connectors
floating-point behavior, compiler flags
Fortran applications, compiler flags
FPM (fast page mode) memory
fragmentation, IP packets
frames (networks)
free list memory management
FSD (Fixed Shroud Device) connectors
fsflush command 2nd
high disk service times and
fstyp command, filesystem information
full-duplex mode
function inlining
function pointers, compiler flags


index_G

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

gateways
gethrtime system call
Gigabit Ethernet
Gigaplane XB crossbar (Sun)
gprof (GNU)
application profile analysis
graded-index fiber
graphics, processor support
grep bufhwm command, UFS buffer cache


index_H

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

half-duplex mode
Hamming code arrays
hard real-time process scheduling
hardware
disks, architecture
failures and error generation
fiberoptic cabling
IDE cables
RAID array management
swap space and
UTP cabling
hardware address
hdparm command
disk benchmarking
disk caches, enabling
IDE performance, enhancing
header length field (TCP headers)
heap space segment (memory)
Heisenbergs Principle of Uncertainty
hertz
high I/O skew (disks)
High Level Interface (HILI) group (Ethernet)
hme driver, Ethernet status, displaying
hmeconfig, Ethernet parameters, changing
host adapters
IPI-3
SCSI
hot spare pools (RAID arrays)


index_I

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

i-cache (processor instructions)
I-series processors (Sun)
I/O (input/output) subsystem
PIO
process accounting
tuning, importance of
UFS cache, disabling
von Neumann model 2nd
I/O tracing
prex command
process filtering
I/O wait time (processor)
IBM PowerPC processors, memory page size
ICMP errors, Message fragmentation needed
IDE (Integrated Drive Electronics)
architecture
compared to SCSI
disk caches
limitations
Linux, enhancing performance
idle time (processor)
IEEE 1394 [See FireWire]
if statements, conditional branches (instruction sets)
IMPs (internet message processors)
in-kernel NFS servers
inbound traffic
indirect blocks (filesystems)
inode cache
NFS
UFS
inodes
UFS (Unix Filesystem)
input/output [See I/O subsystem]2nd [See I/O subsystem]
instruction register
Instruction Set Architecture (ISA) 2nd
instructions
branch prediction
conditional branches
i-cache
instruction set architecture, specifying
LDSTUB
NOP (no-operation)
stages of
variable length, effect on processor
Integrated Drive Electronics [See IDE]
Intel
80x86 processors, memory page size
MMX extensions
P6 processor
Intelligent Peripheral Interface (IPI)
interconnectionless architectures, multiprocessor systems
interfaces
architecture
ATA
ATAPI
ATM (Asynchronous Transfer Mode)
EIDE
Ethernet
signal basics
topologies
FC (Fibre Channel)
FDDI (Fiber Distributed Data Interface)
FireWire
IDE architecture
IPI (Intelligent Peripheral Interface)
networks, promiscuous mode
SASI (Shugart Associates System Interface)
Ultra-ATA
USB (Universal Serial Bus)
interleaving memory
intermediate systems (networks)
internal clock rate
internal transfer speed (disks)
compared to real speed
internet message processors (IMPs)
internetworking, books about
internetworks
interprocess communication (IPC)
interrupts
Linux
round robin distribution
Solaris
intimately shared memory
intimately shared memory (ISM)
inverse request-response patterns (networks)
iostat command
disk bottlenecks, locating
disk performance monitoring
NFS operational statistics
iozone disk benchmark
IP addresses
classes
classless
duplicates
private
subnetting
IP protocol
header fields
packets
fragmentation
time-to-live
protocol field (header)
IPC (interprocess communication)
IPI (Intelligent Peripheral Interface) architecture
IPI-2 string controllers
IPI-3 host adapters
irqtune application, Linux interrupt adjustments
ISA (Instruction Set Architecture) 2nd
isainfo, kernel information
ISM (intimately shared memory)
isochronous data transfers


index_J

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

jacks
JFS (Journaled Filesystem)
journaling
modes, setting
support


index_K

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

keepalive timer (TCP)
kernel
I/O tracing
prex command
process filtering
tuning variables
tuning, single-user workstations
virtual memory variables
web servers
workgroup servers and
write throttle variable
kernel memory allocation errors
kernel threads
categories
dispatch queue
quantum
sleep state
Kesslers Best Bin algorithm
Knuth-Morris-Pratt searching algorithms
kstat unix\:0\:biostats command, UFS buffer cache statistics
kswapd (Linux pagedaemon)
ktrace command, I/O tracing


index_L

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

L1 cache 2nd
L2 cache
late collisions
latency
wide area networks and NFS
LDSTUB (load-store-unsigned-byte) instructions
level 1 cache 2nd
level 2 cache
levels of transformation
lightweight process (LWP), kernel threads
limit command, user limitations
linear mode device (RAID arrays), Linux md package
lines (caches)
Link Integrity test
linked list cache-busting
Linpack benchmark
Linux
bdflush command
disk bottlenecks, locating
elevator algorithms
ext2fs
ext3fs
filesystems, ReiserFS 2nd
IDE performance, enhancing
interrupts
JFS (Journaled Filesystem)
kernel variable tuning
md package, RAID array management
memory page size, determining
process scheduling
processor load, vmstat command
ps command, process memory consumption
swap space, monitoring
TCP buffer settings
virtual memory management
lmbench memory benchmarking tool
load average (processors)
locking, NFS and
lockstat command, mutex information
log-structured filesystem logging
logging filesystems
overview
Solaris
Solstice DiskSuite software
Logical Link Control (LLC) group (Ethernet)
logical unit number (LUN)
logical volumes
loop optimizations
low watermarks (TCP)
LUN (logical unit number)
LWP (lightweight process), kernel threads


index_M

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

MAC (media access control)
MAC address
major page faults
majority consensus algorithm (Solstice DiskSuite)
Manchester encoding
mapping caches
MAR (memory address register)
master ports
MAU (medium attachment unit)
maxcontig filesystem parameter
maximum segment size (MSS), TCP packets
maximum transmission unit (MTU)
maxpgio, kernel variable tuning
maxusers parameter
kernel tables
MB (megabyte)
MBus architecture
md package (Linux RAID array management)
MDR (memory data register)
Mean Time Between Failure (MTBF)
Mean Time To Data Inaccessibility (MTTDI)
Mean Time To Data Loss (MTTDL)
Mean Time To Repair (MTTR)
measurements, rationale for
media access control (MAC)
medium attachment unit (MAU)
megabyte (MB)
megaflops (MFLOPS)
megahertz (MHz)
memory [See also virtual memory]
access time
address space layout
addressing limitations
associative mapping
benchmarking
Linpack benchmark
tools
cache access times
cache lines
caches and hierarchy
characteristics
consumption considerations 2nd
cycle time
data path bits
data transfer measurement methods
demand paging
direct cache mapping
filesystem caching
bdflush command
cyclic caching
disk interaction constraints
fsflush command
priority paging
free list
hardware-based RAID array management
interleaving
intimately shared memory
kernel memory allocation errors
lm benchmark
management, Linux
modules, types of
monitoring
memstat command
sar command
vmstat command
NFS requirements
NVRAM, NFS and
page coloring
page scanner
paging
paging compared to swapping
performance metrics
physical implementation
process accounting
process utilization, displaying
processes
monitoring consumption
ps command
uax command
processor caches
requirements
determining
estimating
speed compared to processor speed
STREAM benchmark
swap space
anonymous memory
monitoring
optimizing
sizing
swapping
effect on interactive performance
system performance, effect on
TLB (transaction lookaside buffer)
uncacheable areas
virtual address spaces
web servers
workgroup servers and
memory address register (MAR)
memory data register (MDR)
memory management unit (MMU)
memory system (von Neumann model)
memstat command, memory monitoring
Message fragmentation needed (ICMP error)
message passing patterns (networks)
metadata logging (filesystems)
metadb command, Solstice DiskSuite state databases
metadevices
metainformation database (Solstice DiskSuite)
metamirrors (Solaris DiskSuite)
metattach command, Solaris DiskSuite
metrics, memory performance
MFLOPS (megaflops)
MHz (megahertz)
microprocessors [See processors]
microSPARC processors, memory page size
microstate accounting
enabling
migrating ext2fs to ext3fs
minfree filesystem parameter
minor page faults
MIPS (millions of instructions per second)
MIPS processors, memory page size
mirrored striping
mirroring disks
disk replacement
Linux md package
Solaris DiskSuite
miss cost (memory cache)
mkinitrd command, RAID arrays, booting
mkraid device command, Linux md package
MMU (memory management unit)
MMX extensions (Intel)
monacct command, process accounting
Motorola PowerPC processors, memory page size
mount command
journaling mode, setting
logging option
options to CacheFS
mounting network filesystems
mounts, NFS (Network File System), nesting
mpstat command
mutex information
processor statistics
MSS (maximum segment size), TCP packets
MTBF (Mean Time Between Failure)
MTTDI (Mean Time To Data Inaccessibility)
MTTDL (Mean Time To Data Loss)
MTTR (Mean Time To Repair)
MTU (maximum transmission unit)
MTU (message transfer unit), TCP packets and
multi-initiator SCSI
multi-word DMA
multimode fiber
multiporting
multiprocessor systems [See also processors; SMP]2nd
application behavior
busses
caches
communication basics 2nd
crossbars
Fireplane (Sun)
interconnectionless architectures
load average statistic
optimum processor load considerations
process queues
processor statistics
mpstat command
thread synchronization
threads
multitasking
mutexes
lockstat command
mpstat command
processor load and
mutual exclusion locks (mutexes)


index_N

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Nagles algorithm 2nd
naive search algorithm
naming services, static performance issues
National Committee on Information Technology Standards (NCITES)
NCA (Network Cache Accelerator)
NCITES (National Committee on Information Technology Standards)
ndd command
Ethernet, changing operational mode
TCP tuning
nesting NFS mounts
netstat, error rate monitoring
network address
network administrator, responsibilities compared to systems administrator
Network Cache Accelerator (NCA)
Network File System [See NFS]
network layer (OSI model)
Network Lock Manager (NLM)
networks
architecture
ATM (Asynchronous Transfer Mode)
books about
bridges
broadcast frames
CDIR (classless interdomain routing)
clients
collisions
data links
duplex modes
FDDI (Fiber Distributed Data Interface)
filesystems, mounting
frames
gateways
interfaces, promiscuous mode
intermediate systems
internetworks
IP
latency, wide area networks and NFS
MAC (media access control)
OSI model architecture
packet size distribution
packets
pattern analysis
payload-free packets
performance factors
physical layer components
cables and connectors
interfaces
repeaters
routers
routing
scalability
servers
sizing, web servers and
subnetting
switches
TCP
three-way handshake
transparent bridges
UDP
unicast traffic
newfs command
cluster size, specifying
free space parameter
rotational delay parameter
NFS (Network File System)
buffer cache
CacheFS
CacheFS and
caches
DNLC (directory name lookup cache)
inode
client behavior
clients
bursty transfers
sequential transfers
disk subsystems, designing
memory requirements
nesting mounts
NVRAM
operating principles
operational statistics
overview
remote swapfiles
rnode cache
Samba and
servers
monitoring with nfsstat
tuning
types of
threads and
version, selecting
wide area networks and latency issues
workgroup servers and
nfsstat command
NFS operational statistics
NFS servers
NLM (Network Lock Manager)
non-unit stride cache-busting
nonuniform memory access (NUMA)
NOP (no-operation) instructions
nsswitch.conf file, static performance issues
NUMA (nonuniform memory access)
NVRAM
NFS
RAID arrays and


index_O

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

OLTP (online transaction processing), benchmarks
operating system
kernel variables, tuning
real-time process scheduling
optical bypass switch (FDDI network interface)
optimization [See performance tuning]
OSI model network architecture
outbound traffic
overhead, process accounting


index_P

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

P6 processor (Intel)
packet-switched protocols
packets 2nd
collisions
flow control
fragmentation
MSS (maximum segment size)
payload-free
size distribution
time-to-live
page scanner
paging
compared to swapping
disk problems and
free list
page coloring
swap space
anonymous memory
sizing
virtual memory
parallel member write policies (disk mirroring)
parameters
caches, changing
Ethernet, changing
maxcontig
maxusers
kernel tables
minfree
pt_cnt
rotational delay
parity context (disk striping)
parity-protected striping
parity-protected striping with independent disks
pattern analysis, networks
data transfer patterns
inverse request-response
message passing patterns
packet size distribution
request-response
payload-free packets
PCI (Peripheral Component Interconnect) bus
architecture
CompactPCI
PCI Special Interest Group web site
PCI-to-PCI bridge
perf file, sar command, enabling
performance accounting
sar command
SE Toolkit software
performance analysis tools
performance instrumentation counters (PICs)
performance tuning
definition
dynamic performance analysis
environment, importance of
Heisenbergs Principle of Uncertainty
I/O subsystem, importance of
latency
problems with, overview of
proxy servers
resource utilization
single-user workstations
filesystems
kernel
swap space
static factors
swap space
test design
throughput and latency considerations
trade-offs and
web servers
disks
filesystems
kernel
memory
network sizing
swap space
workgroup servers
disks
filesystems
kernel
memory
NFS
swap space
Peripheral Component Interconnect bus [See PCI bus]
peripherals
interconnect performance monitoring tools
PCI bus architecture
SBus architecture
persistent superblocks (Linux md package)
physical address
physical layer (OSI model)
cables and connectors
interfaces
PICs (performance instrumentation counters)
PIO (programmed I/O)
pipes, sticky bit
piplining
conditional branches
platters
plugs
pmap command, estimating memory requirements with
point-to-point Fibre Channel topology
political considerations, workload management
PowerPC processors, memory page size
preempting threads
prefetching, application optimization and
presentation layer (OSI model)
prex command
I/O tracing
restarting
priocntl command, process priorities, adjusting
priority paging
priority, processes
adjusting 2nd
determining 2nd
private IP addresses
probe-based application analysis
Solaris TNF
problem solving
algorithms
hardware level architecture
levels of transformation
memory hierarchy and
programming languages and
/proc directory (Linux), kernel variable tuning
process accounting
enabling
overhead
records, reviewing
process filtering, I/O tracing
process monitoring
prstat command
top command
process queues, viewing state
process scheduling
classes
dispatch queues
dispatch tables, tuning
dynamic priority
Linux
priorities
adjusting 2nd
determining 2nd
real-time
scheduling classes, listing
SMP systems
Solaris
Solaris threading model
static priority
processes
blocked
free list memory management
memory
estimating requirements
monitoring consumption
ps command
uax command
memory usage, displaying
virtual address spaces
processors [See also mulitprocessor systems; SMP]
architecture, specifying with compiler options
cache hierarchy
cache mapping
cache organization
cache size anomalies
cache-busters
caching
CISC design
clock rates
conditional branches
CPI (cycles per instruction)
critical path
DEC Alpha
memory page size
design basics
dispatch queues
failure
FISC design
graphics support
I/O wait time
idle time
improvements in
instruction stages
Intel 80x86, memory page size
Intel P6
internal clock rate
load average
load, vmstat command
locality
metrics, MIPS
microSPARC, memory page size
MIPS, memory page size
MMX extensions
mpstat command
optimum load considerations
performance monitoring
performance, importance of
piplining
PowerPC, memory page size
process accounting
process queues
process scheduling
psrinfo command
RISC design, second generation
sets, creating with psrset command
SPARC
LDSTUB instructions
memory page size
speed compared to memory speed
spin locking
state, changing with psradm command
Sun E-series
Sun I-series
superpiplined
superscalar 2nd
SuperSPARC, memory page size
system time
system time compared to user time
time command operation
Transmeta Crusoe 2nd
UltraSPARC, memory page size
user time
variable length instructions
VIS extensions
vmstat command
von Neumann model
profile feedback (compilers)
profiler-based application analysis
GNU gprof
profiling
application analysis
compiling with
executing with
program counter
programmed I/O (PIO)
programming code [See also applications]
algorithms
Boyer-Moore searching
Knuth-Morris-Pratt searching
string searching
analyzing
ptime command
time command
timex command
arithmetic optimization
compiler optimization considerations
compilers, profile feedback
fast compiler option
linked list cache-busting
loop optimization
microstate accounting, enabling
non-unit stride cache-busting
pitfalls
poor, reasons for
string optimization
TICK register, enabling
timing information, gethrtime system call
xarch compiler option
xchip compiler option
xcrossfile compiler option
xdepend compiler option
xinlining compiler option
xO compiler option
xprefetch compiler option
xsfpconst compiler option
xvector compiler option
programming languages, problem solving and
promiscuous mode (network interfaces)
protocol field (IP headers)
protocols
circuit-switched
IP
NLM (Network Lock Manager)
packet-switched
routing
TCP
three-way handshake
UDP
proxy servers, tuning
prstat command, process monitoring
prtconf command, synchronous transfer rate (SCSI)
ps command
process memory consumption
process priorities, determining 2nd
psradm command, processor state, changing
psrinfo command
processor failure monitoring
processor information
psrset command, processor sets, creating
pt_cnt parameter
ptime command
application analysis
disk write throttle


index_Q

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

QoS (quality of service) (ATM)
quantum (kernel threads)


index_R

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

RAID
arrays
creating with Linux md package
design considerations
autodetection
booting
deployment models
attribute-intensive home directories
data-intensive home directories
database environment
high performance computing
hardware-based array management
hot spare pools
large I/O applications
level 5
Linux md package
Solaris DiskSuite and
levels
distributed parity-protected striping
Hamming code arrays
mirrored striping
mirroring
overlapping
overview
parity-protected striping
parity-protected striping with independent disks
selecting
striping
Linux md package
logical volumes
MTBF (Mean Time Between Failure)
MTTDI (Mean Time To Data Inaccessibility)
MTTDL (Mean Time To Data Loss)
MTTR (Mean Time To Repair)
NVRAM and
software-based array management
Solstice DiskSuite
strip unit size, selecting
raidstart command
raidstop command
random access (disks)
RAID disk striping
storage capacity considerations
rate metric (SPECfp_rate95)
RDRAM (Direct Rambus)
read latency rate
read/write heads
reads (disks)
real-time scheduling
dispatch table
receive buffer (TCP)
refactored code
register file
registers, PCI busses
ReiserFS
overview
tail packing
renice command, process priority, adjusting 2nd
repeaters
replicas (Solstice DiskSuite state databases)
request-response patterns (networks)
resolv.conf file, static performance issues
resource utilization
resources
books
application performance
benchmarking
computer architecture
Java code optimization
networking
routing
TCP
Unix
workload management
environment, limiting users
management of
utilization, process accounting
resynchronization (disk mirroring)
retransmission (TCP)
rings (FDDI network interface)
RISC (reduced instruction set computing)
branch delay slots
instruction set length
second generation design
RJ-11 connectors
RJ-45 connectors
rnode cache, NFS
rnodes (filesystems)
rotational delay filesystem parameter
rotational position sense (RPS), misses
round robin distribution (interrupts)
routers
routing
books about
protocols
RPS (rotational position sense), misses
runacct command, process accounting


index_S

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Samba, NFS and
sar command
disk partition statistics
enabling
inode cache monitoring
memory monitoring
NFS buffer cache
records, reviewing
swap space monitoring
SASI (Shugart Associates System Interface)
SBus (Sun)
architecture
burst transfer size
clock speed
interrupt processing
transfer mode
SC connectors
scalability, networks
scheduling classes
listing
SCSI (Small Computer System Interface)
bus transactions
bus utilization
command queuing
compared to IDE
different speed devices, mixing
differential signaling
disk caches
implementations
multi-initiator
synchronous compared to asynchronous transfers
termination
SCSI-3
SDRAM (synchronous DRAM)
SE Toolkit software
Second Extended Filesystem (ext2fs)
sectors
seek time (disks), reducing
segments, virtual memory
send buffer (TCP)
sequence number (TCP headers)
sequential access (disks)
RAID disk striping
serial member write policies (disk mirroring)
serial-optical coupler (SOC), Fibre Channel bus connections
servers
NFS
monitoring with nfsstat
tuning
types of
proxy
TCP/IP
web
disks
filesystems
kernel
memory
network sizing
swap space
workgroup
disks and
filesystems and
kernel and
memory and
NFS and
swap space and
session layer (OSI model)
shells, user limits on resources
Shugart Associates System Interface (SASI)
signal degradation, repeaters and
single-mode fiber (SMF)
size of
slave ports
sleep state (threads)
slow start algorithm (TCP)
Small Computer System Interface [See SCSI]
SMF (single-mode fiber)
SMP (symmetric mulitprocessor) systems, process scheduling
SOC (serial-optical coupler), Fibre Channel bus connections
soft real-time process scheduling
software [See also applications]
irqtune, Linux interrupt adjustments
Linux md package
RAID array management
SE Toolkit
Solstice DiskSuite 2nd
Solaris
cyclic caching
disk bottlenecks, locating
disk quotas, enabling
filesystem logging
filesystems, tmpfs
fsflush command
interrupts
kernel variable tuning
memory addressing limitations
memory page size, determining
memory requirements, determining
page coloring algorithms
performance accounting software
priority paging
process accounting, enabling
process scheduling
processor load, vmstat command
remote swapfiles
swap space, displaying status
swapfiles, creating
threading model
TNF
argument types
inserting probes
uax command, process memory consumption
ufs write throttling
vnodes
Solstice DiskSuite
filesystem logging
RAID array management
SPARC processors
LDSTUB instructions
memory page size
SPARC V9 instruction set
spatial locality, caches
SPEC benchmarks
SPECweb99 benchmark
speed optimized RISC processors
speed, attitude and requirements in modern world
spin locking
spindle
spindle synchronization 2nd
SRAM (static random access memory)
ST connectors
stacks
stalling (piplining)
state database (Solstice DiskSuite)
static interrupt distribution (Solaris)
static performance tuning
hardware failures and
static priority
static random access memory (SRAM)
step-index multimode fiber
sticky bit
storage-area networking
store-and-forward switches
STREAM memory benchmarking tool
strength reduction (arithmetic optimization)
string controllers, IPI-2
string optimization
string searching algorithms
string utilization, SCSI
stripe unit size, selecting
stripe width (RAID arrays)
striping disks
Linux md package
Solaris DiskSuite
submirrors
submirrors (RAID arrays)
subnet masks
subnetting
alternatives to
Sun
E-series processors
Fireplane
Gigaplane XB crossbar
I-series processors
MBus architecture
microSPARC processors, memory page size
SBus
architecture
burst transfer size
clock speed
interrupt processing
transfer mode
Solstice DiskSuite software 2nd
SPARC processors, memory page size
SuperSPARC processors, memory page size
UltraSPARC processors, memory page size
UltraSPARC-III architecture
VIS extensions
XBus architecture
superpipelined processors
superscalar processors 2nd
SuperSPARC processors, memory page size
swapfiles, creating (Solaris)
swapping
compared to paging
disk problems and
swap space
anonymous memory
monitoring
optimizing
sizing
tuning, single-user workstations
web servers
workgroup servers and
Unix mechanism
switches (networks)
SYN bit (TCP)
SYN flooding (denial-of-service attacks)
synchronous data transfers, SCSI
synchronous DRAM (SDRAM)
synchronous writes (disks)
system file, persisting maxpgio value changes
system time (processor)
System V model (process scheduling)
systems administrator, responsibilities compared to network administrator


index_SYMBOL

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

100BASE-T4 Ethernet
100BASE-TX Ethernet
10BASE-T Ethernet
10BASE2 Ethernet cabling
10BASE5 Ethernet cabling
5-4-3 rule (Ethernet)
64-bit architecture
applications
disadvantages
80x86 processors (Intel), memory page size


index_T

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

T10 Technical Committee web site
T11 Technical Committee web site 2nd
T13 Technical Committee web site
tags, cache mapping
tail packing (ReiserFS)
target identifier, SCSI addresses
TCP (Transmission Control Protocol)
books about
compared to UDP
congestion window
data transfer rates
deferred acknowledgements
denial-of-service attacks, SYN flooding
FIN_WAIT_2 timer
headers
keepalive timer
low watermarks
Nagles algorithm
receive buffers
retransmission
send buffer
slow start algorithm
three-way handshake
TIME_WAIT timer
transmit high watermark
tuning with ndd command
window scaling
windows
TCP/IP
clients
servers
temporal locality, caches
terminal-driven database benchmarks
termination, SCSI
testing
design considerations
kernel variable tuning, importance of
Third Extended Filesystem (ext3fs)
thrashing
threads
barriers
categories
critical sections
dispatch queue
mulitprocessor systems
mutexes
NFS and
preempting
quantum
sleep state
Solaris model
spin locking
synchronization, mulitprocessor systems
three-way handshake (TCP)
TICK register, enabling
time command, application analysis
time-to-live (IP packets)
TIME_WAIT timer (TCP)
timeshare class kernel threads
timex command, application analysis
tiobench disk benchmark
TLB (transaction lookaside buffer)
tmpfs filesystem
TNF (trace normal form)
argument types
inserting probes
Token Bus
Token Ring
tools, performance analysis
top command, process monitoring
TP-DDI (Twisted Pair Distributed Data Interface )
TPC (Transaction Processing Council), benchmarks
tracks
transaction lookaside buffer (TLB)
Transmeta Crusoe processor 2nd
Transmission Control Protocol [See TPC]
transmit high watermark (TCP)
transparent bridges
transport layer (OSI model)
tunefs command
tuning [See performance tuning]
Twisted Pair Distributed Data Interface (TP-DDI)


index_U

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

uax command, process memory consumption
UDP (User Datagram Protocol)
compared to TCP
UFS (Unix Filesystem)
buffer cache
caching
cluster size
inode cache
inodes
overview
ufs write throttling (disks)
ulimit command, user limits
Ultra-ATA
UltraDMA
UltraSPARC Port Architecture switched interconnect (UPA)
UltraSPARC processors, memory page size
UltraSPARC-III
architecture
compiler flags
UMA (uniform memory access)
unicast network traffic
unit stride, programming code and
Universal Serial Bus (USB)
Unix
books about
memory swapping
Unix Filesystem [See UFS]
unswitched networks, resolving collision problems
UPA (UltraSPARC Port Architecture switched interconnect)
uptime command, processor load average
usage and performance agreements
USB (Universal Serial Bus)
User Datagram Protocol (UDP)
user defined benchmarks
automation considerations
problem set selection
rule creation
runtime selection
user time (processor)
user-space NFS servers
users
disk quotas
educating
environmental limits
limitations on
supported, setting number of
usage and performance agreements
utilization, resources
UTP (unshielded twisted-pair) cabling


index_V

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

variable length subnet mask (VLSM)
VAX MIPS
very long instruction word (VLIW)
virtual address space, processes
virtual memory [See also memory]
architecture
kernel variables
management, Linux
pages
segments
VIS extensions (Sun)
VLIW (very long instruction word)
VLSM (variable length subnet mask)
vmstat command
DNLC (directory name lookup cache)
intervals for
memory monitoring
memory requirements, determining
process queues
processor load
processor statistics
vnodes
von Neumann model
VUPs (VAX units of processing)
VxFS filesystem


index_W

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

watermarks
disk write throttling
TCP
web performance, benchmarking
web servers
disks
filesystems
kernel
memory
network sizing
swap space
web sites
hdparm command
JFS
lmbench memory benchmark
PCI Special Interest Group
SPEC benchmarks
STREAM benchmark
T10 Technical Committee
T11 Technical Committee 2nd
T13 Technical Committee
tiobench
TPC (Transaction Processing Council)
VxFS filesystem
wide area networks, NFS and latency issues
window scaling (TCP)
windowing, TCP packet flow control
workflow characterization
performance analysis tools
process accounting
workload management
workgroup servers
disks
filesystems
kernel
memory and
NFS
swap space
workload management
compared to financial transaction management
guidelines
maxusers parameter
political considerations
usage and performance agreements
users
limitations on
limiting resource environment
workstations (single-user)
filesystem tuning
kernel tuning
swap space tuning
world wide name (WWN)
write latency rate
write throttling (disks)
write-anywhere filesystems
write-back behavior (caches)
write-through behavior (caches)
writes (disks)
WWN (world wide name)


index_X

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

XBus architecture


index_Z

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

ZBR (zone bit rate recording)
zone bit rate (ZBR) recording


Chapter 1. An Introduction to Performance Tuning

There are a thousand hacking at the branches of evil to one who is striking at the root.

—Henry David Thoreau, 1854

Going faster seems to be a central part of human development. As little as three hundred years ago, the fastest you could expect to go was a few tens of miles an hour, aboard a fast clipper ship with a stiff wind behind you. Now, we must expand our view to things such as the fastest achievable speed while remaining in the Earth's atmosphere -- perhaps fifteen hundred miles an hour, twice the speed of sound, if you are a civilian without access to the latest, fastest military aircraft. The journey that used to take three weeks under sail from London to New York City now takes us as little as two and a half hours, sipping champagne the whole way.

This innate human desire to go fast is expressed in many ways: microwave ovens let us cook dinner quickly, high-performance automobiles and motorcycles give us a wonderful thrill, email lets us communicate at almost the speed of thought. But what happens when that email server is overwhelmed, as when we all log in at eight o'clock in the morning to check what has gone on while we've slept? Or when the procurement system for the company that distributes microwave ovens is only able to handle half of the workload, or when a mechanical engineer's CAD system runs so slowly that the car engine she's designing won't be ready in time for the new model year?

These are the problems facing us in performance tuning: it is the adaptation of the speed of a computer system to the speed requirements imposed by the real world.

Sometimes these problems are subtle, sometimes not. We must carefully consider the changes in the system that caused it to become unacceptably slow. These influences might come from within, such as a heavier load placed on the system, or from without, such as a new operating system revision subtly changing -- or utterly replacing -- an algorithm for resource management that is critically important to our software. The solutions are sometimes quick (a matter of adjusting one "knob" slightly), or sometimes slow and painful (a task for weeks of analysis, consultations with vendors, and careful redesign of an infrastructure).

When I originally started work on this book, it seemed straightforward: take Mike Loukides's excellent but dated first edition and overhaul it for modern computer systems. As I progressed further along, I realized that this book is actually about much more than simply performance tuning. It really covers two distinct areas:

Performance tuning

The art of increasing performance for a specific application set (also known as "squeezing blood from a stone").

Capacity planning

Deciding what hardware to purchase to fulfill a given role (also known as "fortune telling").

These areas are all underpinned by the science of computer architecture. This book does not concentrate on application design; rather, it focuses on the operating system, the underlying hardware, and their interactions.

To most systems administrators, a computer is really a black box. This is perfectly reasonable for many tasks: after all, it's certainly not necessary to understand how the operating system manages free memory to configure and maintain a mail server. However, in performance tuning -- which is, at heart, very much about the underlying hardware and how it is abstracted -- truly understanding the behavior of the system involves a detailed knowledge of the inner workings of the machine. In this chapter, we'll briefly discuss some of the most important concepts of computer architecture, and then go into the fundamental principles of performance tuning.


1.1 An Introduction to Computer Architecture

A full discussion of computer architecture is far beyond the level of this text. Periodically, we'll go into architectural matters, in order to provide the conceptual underpinnings of the system under discussion. However, if this sort of thing interests you, there are a great many excellent texts on the topic. Perhaps the most commonly used are two textbooks by John Hennessy and David Patterson: they are titled Computer Organization and Design: The Hardware/Software Interface and Computer Architecture: A Quantitative Approach (both published by Morgan Kaufmann).

In this section, we'll focus on the two most important general concepts of architecture: the general means by which we approach a problem (the levels of transformation), and the essential model around which computers are designed.

1.1.1 Levels of Transformation

When we approach a problem, we must reduce it to something that a computer can understand and work with: this might be anything from a set of logic gates, solving the fundamental problem of "How do we build a general-purpose computing machine?" to a few million bits worth of binary code. As we proceed through these logical steps, we transform the problem into a "simpler" one (at least from the computer's point of view). These steps are the levels of transformation.

1.1.1.1 Software: algorithms and languages

When faced with a problem where we think a computer will be of assistance, we first develop an algorithm for completing the task in question. Algorithms are, very simply, a repetitive set of instructions for performing a particular task -- for example, a clerk inspecting and routing incoming mail follows an algorithm for how to properly sort the mail.

This algorithm must then be translated by a programmer into a program written in a language.[1] Generally, this is a high-level language, such as C or Perl, although it might be a low-level language, such as assembler. The language layer exists to make our lives easier: the structure and grammar of high-level languages lets us easily write complex programs. This high-level language program, which is usually portable between different systems, is then transformed by a compiler into the low-level instructions required by a specific system. These instructions are specified by the Instruction Set Architecture.

[1] It has been conjectured that mathematicians are devices for transforming coffee into theorems. If this is true, then perhaps programmers are devices for transforming caffeine and algorithms into source code.

1.1.1.2 The Instruction Set Architecture

The Instruction Set Architecture, or ISA, is the fundamental language of the microprocessor: it defines the basic, indivisible instructions that we can execute. The ISA serves as the interface between software and hardware. Examples of instruction set architectures include IA-32, which is used by Intel and AMD CPUs; MIPS, which is implemented in the Silicon Graphics/MIPS R-series microprocessors (e.g., the R12000); and the SPARC V9 instruction set used by the Sun Microsystems UltraSPARC series.

1.1.1.3 Hardware: microarchitecture, circuits, and devices

At this level, we are firmly in the grasp of electrical and computer engineering. We concern ourselves with functional units of microarchitecture and the efficiency of our design. Below the microarchitectural level, we worry about how to implement the functional units through circuit design: the problems of electrical interference become very real. A full discussion of the hardware layer is far beyond us here; tuning the implementations of microprocessors is not something we are generally able to do.

1.1.2 The von Neumann Model

The von Neumann model has served as the basic design model for all modern computing systems: it provides a framework upon which we can hang the abstractions and flesh generated by the levels of transformation.[2] The model consists of four core components:

[2] A good book to read more about the von Neumann model is William Aspray's John von Neumann and the Origins of Modern Computing (MIT Press).

  • A memory system, which stores both instructions and data. This is known as a stored program computer. This memory is accessed by means of the memory address register (MAR), where the system puts the address of a location in memory, and a memory data register (MDR), where the memory subsystem puts the data stored at the requested location. I discuss memory in more detail in Chapter 4.

  • At least one processing unit, often known as the arithmetic and logic unit (ALU). The processing units are more commonly called the central processing unit (CPU).[3] It is responsible for the execution of all instructions. The processor also has a small amount of very fast storage space, called the register file. I discuss processors in detail in Chapter 3.

    [3] In modern implementations, the "CPU" includes both the central processing unit itself and the control unit.

  • A control unit, which is responsible for controlling cross-component operations. It maintains a program counter, which contains the next instruction to be loaded, and an instruction register, which contains the current instruction. The peculiarities of control design are beyond the scope of this text.

  • The system needs a nonvolatile way to store data, as well as ways to represent it to the user and to accept input. This is the domain of the input/output (I/O) subsystem. This book primarily concerns itself with disk drives as a mechanism for I/O; I discuss them in Chapter 5. I also discuss network I/O in Chapter 7.

Despite all the advances in computing over the last sixty years, they all fit into this framework. That is a very powerful statement: despite the fact that computers are orders of magnitude faster now, and being used in ways that weren't even imaginable at the end of the Second World War, the basic ideas, as formulated by von Neumann and his colleagues, are still applicable today.

1.1.3 Caches and the Memory Hierarchy

As you'll see in Section 1.2 later in this chapter, one of the principles of performance tuning is that there are always trade-offs. This problem was recognized by the pioneers in the field, and we still do not have a perfect solution today. In the case of data storage, we are often presented with the choice between cost, speed, and size. (Physical parameters, such as heat dissipation, also play a role, but for this discussion, they're usually subsumed into the other variables.) It is possible to build extremely large, extremely fast memory systems -- for example, the Cray 1S supercomputer used very fast static RAM exclusively for memory.[4] This is not something that can be adapted across the spectrum of computing devices.

[4] Heat issues with memory was the primary reason that the system was liquid cooled. The memory subsystem also comprised about three-quarters of the cost of the machine in a typical installation.

The problem we are trying to solve is that storage size tends to be inversely proportional to performance, particularly relative to the next highest level of price/performance. A modern microprocessor might have a cycle time measured in fractions of a nanosecond, while making the trip to main memory can easily be fifty times slower.

To try and work around this problem, we employ something known as the memory hierarchy. It is based on creating a tree of storage areas (Figure 1-1). At the top of the pyramid, we have very small areas of storage that are exceedingly fast. As we progress down the pyramid, things become increasingly slow, but correspondingly larger. At the foundation of the pyramid, we might have storage in a tape library: many terabytes, but it might take minutes to access the information we are looking for.

Figure 1-1. The memory hierarchy

From the point of view of the microprocessor, main memory is very slow. Anything that makes us go to main memory is bad -- unless we're going to main memory to prevent going to an even slower storage medium (such as disk).

The function of the pyramid is to cache the most frequently used data and instructions in the higher levels. For example, if we keep accessing the same file on tape, we might want to store a temporary copy on the next fastest level of storage (disk). We can similarly store a file we keep accessing from disk in main memory, taking advantage of main memory's substantial performance benefit over disk.

1.1.4 The Benefits of a 64-Bit Architecture

Companies that produced computer hardware and software often make a point of mentioning the size of their systems' address space (typically 32 or 64 bits). In the last five years, the shift from 32-bit to 64-bit microprocessors and operating systems has caused a great deal of hype to be generated by various marketing departments. The truth is that although in certain cases 64-bit architectures run significantly faster than 32-bit architectures, in general, performance is equivalent.

1.1.4.1 What does it mean to be 64-bit?

The number of "bits" refers to the width of a data path. However, what this actually means is subject to its context. For example, we might refer to a 16-bit data path (for example, UltraSCSI). This means that the interconnect can transfer 16 bits of information at a time. With all other things held constant, it would be twice as fast as an interconnect with a 8-bit data path.

The "bitness" of a memory system refers to how many wires are used to transfer a memory address. For example, if we had an 8-bit path to the memory address, and we wanted the 19th location in memory, we would turn on the appropriate wires (1, 2, and 5; we derive this from writing 19 in binary, which gives 00010011 -- everywhere there is a one, we turn on that wire). Note, however, that since we only have 8 bits worth of addressing, we are limited to 64 (28) addresses in memory. 32-bit systems are, therefore, limited to 4,294,967,296 (232) locations in memory. Since memory is typically accessible in 1-byte blocks, this means that the system can't directly access more than 4 GB of memory. The shift to 64-bit operating systems and hardware means that the maximum amount of addressable memory is about 16 petabytes (16777216 GB), which is probably sufficient for the immediately forseeable future.

Unfortunately, it's often not quite this simple in practice. A 32-bit SPARC system is actually capable of having more than 4 GB of memory installed, but, in Solaris, no single process can use more than 4 GB. This is because the hardware that controls memory management actually uses a 44-bit addressing scheme, but the Solaris operating system can only give any one process the amount of memory addressable in 32 bits.

1.1.4.2 Performance ramifications

The change from 32-bit to 64-bit architectures, then, expanded the size of main memory and the amount of memory a single process can have. An obvious question is, how did applications benefit from this? Here are some kinds of applications that benefitted from larger memory spaces:

  • Applications that could not use the most time-efficient algorithm for a problem because that algorithm would use more than 4 GB of memory.

  • Applications where caching large data sets is critically important, and therefore the more memory available to the process, the more can be cached.

  • Applications where the system is short on memory due to overwhelming utilization (many small processes). Note that in SPARC systems, this was not a problem: each process could only see 4 GB, but the system could have much more installed.

In general, the biggest winners from 64-bit systems are high-performance computing and corporate database engines. For the average desktop workstation, 32 bits is plenty.

Unfortunately, the change to 64-bit systems also meant that the underlying operating system and system calls needed to be modified, which sometimes resulted in a slight slowdown (for example, more data needs to be manipulated during pointer operations). This means that there may be a very slight performance penalty associated with running in 64-bit mode.


1.2 Principles of Performance Tuning

In this book, I present a few "rules of thumb." As an old IBM technical bulletin says, rules of thumb come from people who live out of town and who have no production experience.[5]

[5] MVS Performance Management, GG22-9351-00.

Keeping that in mind, I've found that much of the conceptual basis behind system performance tuning can be summarized in five principles: understand your environment, nothing is ever free, throughput and latency are distinct measurements, resources should not be overutilized, and one must design experiments carefully.

1.2.1 Principle 0: Understand Your Environment

If you don't understand your environment, you can't possibly fix the problem. This is why there is such an emphasis on conceptual material throughout this text: even though the specific details of how an algorithm is implemented might change, or even the algorithm itself, the abstract knowledge of what the problem is and how it is approached is still valid. You have a much more powerful tool at your disposal if you understand the problem than if you know a solution without understanding why the problem exists.

Certainly, at some level, problems are well beyond the scope of the average systems administrator: tuning network performance at the level discussed here does not really require an in-depth understanding of how the TCP/IP stack is implemented in terms of modules and function calls. If you are interested in the inner details of how the different variants of the Unix operating system work, there are a few excellent texts that I strongly suggest you read: Solaris Kernel Internals by Richard McDougall and James Mauro, The Design of the UNIX Operating System by Maurice J. Bach, and Operating Systems, Design and Implementation by Andrew Tanenbaum and Andrew Woodfull (all published by Prentice Hall).

The ultimate reference, of course, is the source code itself. As of this writing, the source code for all the operating systems I describe in detail (Solaris and Linux) is available for free.

1.2.2 Principle 1: TANSTAAFL!

TANSTAAFL means There Ain't No Such Thing As A Free Lunch.[6] At heart, performance tuning is about making trade-offs between various attributes. This usually consists of a list of three desirable attributes -- of which we can pick only two.

[6] With apologies to Robert A. Heinlein's "The Moon Is A Harsh Mistress."

One example comes from tuning the TCP network layer, where Nagle's algorithm provides a way to sacrifice latency, or the time required to deliver a single packet, in exchange for increased throughput, or how much data can be effectively pushed down the wire. (We'll discuss Nagle's algorithm in greater detail in Section 7.4.2.8.)

This principle often necessitates making real, significant, and difficult choices.

1.2.3 Principle 2: Throughput Versus Latency

In many ways, systems administrators who are evaluating computer systems are often like adolescent males evaluating cars. This is unfortunate. In both cases, there are a certain set of metrics, and we try to find the highest value for the most "important" metric. Typically, these are "bulk data throughput" for computers and "horsepower" for cars.

The lengths to which some people will go to obtain maximum horsepower from their four-wheeled vehicles are often somewhat ludicrous. A slight change in perspective would reveal that there are other facets to the performance gem. A lot of effort is spent in optimizing a single thing that may not actually be a problem. I would like to illustrate this by means of a simplistic comparison (please remember that we are comparing performance alone). Vehicle A puts out about 250 horsepower, whereas Vehicle B only produces about 95. One would assume that Vehicle A exhibits significantly "better" performance in the real world than Vehicle B. The astute reader may ask what the weights of the vehicles involved are: Vehicle A weighs about 3600 pounds, whereas Vehicle B weighs about 450. It is then apparent that Vehicle B is actually quite a bit faster (0-60 mph in about three and a half seconds, as opposed to Vehicle A's leisurely five and a half seconds). However, when we compare how fast the vehicles actually travel in crowded traffic down Highway 101 (which runs down the San Francisco Peninsula), Vehicle B wins by an even wider margin, since motorcycles in California are allowed to ride between the lanes ("lane splitting").[7]

[7] For the curious, Vehicle A is an Audi S4 sedan (2.7 liter twin-turbocharged V6). Vehicle B is a Honda VFR800FI Interceptor motorcycle (781cc normally-aspirated V4). Both vehicles are model year 2000.

Perhaps the most often neglected consideration in this world of tradeoffs is latency. For example, let's consider a fictional email server application -- call it SuperDuperMail. The SuperDuperMail marketing documentation claims that it is capable of handling over one million messages a hour. This might seem pretty reasonable: this rate is far faster than most companies need. In other words, the throughput is generally good. A different way of looking at the performance of this mail server would be to ask how long it takes to process a single message. After some pointed questions to the SuperDuperMail marketing department, they reveal that it takes half an hour to process a message. This appears contradictory: it would seem to indicate that the software can process at most two messages an hour. However, it turns out that the SuperDuperMail product is based on a series of bins internally, and moves messages to the next bin only when that bin is completely full. In this example, despite the fact that throughput is acceptable, the latency is horrible. Who wants to send a message to the person two offices down and have it take half an hour to get there?

1.2.4 Principle 3: Do Not Overutilize a Resource

Anyone who has driven on a heavily-travelled highway has seen a very common problem in computing emerge in a different area: the "minimum speed limit" signs often seem like a cruel joke! Clearly, there are many factors that go into designing and maintaining an interstate, particularly one that is heavily used by commuters: the peak traffic is significantly different than the average traffic, funding is always a problem, etc. Furthermore, adding another lane to the highway, assuming that space and funding is available, usually involves temporarily closing at least one lane of the active roadway. This invariably frustrates commuters even more. The drive to provide "sufficient" capacity is always there, but the barriers to implementing change are such that it often takes quite a while to add capacity.

In the abstract, the drive to expand is strongest when collapse is imminent or occuring. This principle usually makes sense: why should we build excess capacity when we aren't fully using what we have? Unfortunately, there are some cases where complete utilization is not optimal. This is true in computing, yet people often push their systems to 100% utilization before considering upgrades or expansion.

Overutilization is a dangerous thing. As a general rule of thumb, something should be no more than 70% busy or consumed at any given time: this will provide a margin of safety before serious degradation occurs.

1.2.5 Principle 4: Design Tests Carefully

I would like to explain this principle by means of a simple example of transferring a 130 MB file over Gigabit Ethernet using ftp.

% pwd

/home/jqpublic

% ls -l bigfile

-rw-------   1 jqpublic staff    134217728 Jul 10 20:18 bigfile

% ftp franklin

Connected to franklin.

220 franklin FTP server (SunOS 5.8) ready.

Name (franklin:jqpublic): jqpublic

331 Password required for jqpublic.

Password: <secret>

230 User jqpublic logged in.

ftp> bin

200 Type set to I.

ftp> prompt

Interactive mode off.

ftp> mput bigfile

200 PORT command successful.

150 Binary data connection for bigfile (192.168.254.2,34788).

226 Transfer complete.

local: bigfile remote: bigfile

134217728 bytes sent in 13 seconds (9781.08 Kbytes/s)

ftp> bye

221 Goodbye.

If you saw this sort of performance, you would probably be very upset: you were expecting to see a transfer rate on the order of 120 MB per second! Instead, you are barely getting a tenth of that. What on earth happened? Is there a parameter that's not set properly somewhere, did you get a bad network card? You could waste a lot of time searching for the answer to that question. The truth is that what you probably measured was either how fast you can push data out of /home, or how fast the remote host could accept that data.[8] The network layer is not the limiting factor here; it's something entirely different: it could be the CPU, the disks, the operating system, etc.

[8] Which is not necessarily the same as "How fast can this data be written to disk?" although it might be.

A great deal of this book's emphasis on explaining how and why things work is so that you can design experiments that enable you to measure what you think you are.

There is a huge amount of hearsay regarding performance analysis. Your only hope of making sense of it is to understand the issues, design tests, and gather data.

The moral of this example is to think very, very carefully when you design performance measurement experiments. All sorts of things might be going on just underneath the surface of the measurement that are causing you to measure something that has little bearing on what you're actually interested in measuring. If you really want to measure something, try and find a tool written specifically to test that component. Even then, be careful: it's very easy to get burnt.

Tuning Kernel Variables

Throughout this book, we will refer to making adjustments to specific kernel variables. Never perform kernel tuning on a live system if you can possibly help it; try and replicate the problem in a controlled laboratory environment and experiment with the tunables there. Even if you do have to make changes on a live system, test your changes extensively.

In Solaris, you can tune kernel variables on the fly with adb, or have them automatically set at boot time. We'll use maxpgio as an example (you can learn more about maxpgio in Section 4.2.5). First, you need to know whether you are booted into a 64-bit or 32-bit kernel. You can find this out by running isainfo -v. Here's an example from a 32-bit system:

% isainfo -v

32-bit sparc applications

And from a 64-bit system:

% isainfo -v

64-bit sparcv9 applications

32-bit sparc applications

In order to check the setting of a kernel variable, start up adb in kernel mode (-k). To view the contents of a variable on a 32-bit system, type the variable name followed by /D; on a 64-bit system, follow the variable name with /E. If you follow the variable name with an incorrect suffix, you will get meaningless output. Here's an example from a 32-bit system:

# adb -k

physmem 3bd6

maxpgio/D

maxpgio:

maxpgio:        40

And from a 64-bit system:

# adb -k

physmem 1f102

maxpgio/E

maxpgio:

maxpgio:        40

If you decide that you need to change the value of this variable, you can then make an immediate change to the variable. Drop out of adb by typing Ctrl-D, and restart it with adb -kw. To change the variable, you must again specify a suffix: use /W on 32-bit systems and /Z on 64-bit systems. This string should then be followed by 0t and the decimal value you wish to set the variable to.

For example, on a 32-bit system, let's change the value of maxpgio to 100:

# adb -kw

physmem 3bd6

maxpgio/W 0t100

maxpgio:        0x28            =       0x64

maxpgio/D

maxpgio:

maxpgio:        100

We can perform the same change on a 64-bit system:

# adb -kw

physmem 1f102

maxpgio/Z 0t100

maxpgio:        28              =       64

maxpgio/E

maxpgio:

maxpgio:        100

This change will not persist past a reboot. If you want to make the change permanent, edit /etc/system:

*

* change kernel variable at boot-time: 08-11-2001 by gdm

set maxpgio=100

In Linux, depending on whether the variable has been exposed in the /proc tree or not, you have two choices: you can either edit the kernel source code and build a new kernel with your changes, or you can edit the appropriate file in /proc. Editing files in proc is straightforward -- for example, let's say you wish to change the value of the min_percent variable in Linux 2.2, which controls the minimum percentage of system memory available for caching. This variable can be found in the /proc/sys/vm/buffermem file, and the format of that file is min_percent max_percent borrow_percent. We can change the value of min_percent quite straightforwardly:

# cat /proc/sys/vm/buffermem

2 10 60

# echo "5 10 60" > /proc/sys/vm/buffermem

# cat /proc/sys/vm/buffermem

5 10 60

These changes won't persist past a reboot, but you can put an entry in a startup script to perform the tuning automatically.

Again—be careful, and always test your changes!


1.3 Static Performance Tuning

This book largely focuses on the performance tuning of systems in dynamic conditions. I focus almost exclusively on how the system's performance corresponds to the stresses placed on the system. Another sort of performance tuning exists, however, that focuses on static (that is, workload-independent) factors. These factors tend to degrade system performance no matter what the workload is, and are not generally tied to resource contention problems.

By far the largest culprit in static performance issues are problems with naming services: the means by which information is retrieved about an entity. Examples of naming services are NIS+, LDAP, and DNS.

The symptoms are often vague: logins are slow, the windowing environment or a web browser feels sluggish, a new window locks at startup, or the X or CDE login screen hangs. Here are a few places to check, in approximate order of likelihood:

/etc/nsswitch.conf

This file is read only once for each process that is using a naming service, so a reboot may be necessary to make changes take effect.

/etc/resolv.conf

Are the nameservers and the domain specified correctly? Incorrect or lengthy (having many subcomponents) domain specifications may cause DNS to generate a great many requests. The nameservers should be sorted by latency of response.

Is the name service cache daemon (nscd) running?

This process caches name service-provided information for a significant boost to performance. This daemon is standard-issue on Solaris, and available for Linux. Historically, nscd has been blamed for many problems. Most of these problems have been long since fixed: you should probably run it. One exception is when troubleshooting, however, as it can mask underlying name service problems.

Another likely culprit is the Network File System (NFS). It is important to not nest mounts. For example, let's say my workstation mounts three directories from three distinct NFS servers:

alpha:/home

bravo:/home/projects

delta:/home/projects/system-performance-tuning-2nd-ed

Reading the file /home/projects/system-performance-tuning-2nd-ed/README requires all three NFS servers to be up, and generates quite a bit of excess traffic. It would be much faster to have separate mount points for everything.

1.3.1 Other Miscellaneous Things to Check

Duplicate IP addresses on a network, as well as many other host interface misconfigurations, can cause all sorts of problems. Try to maintain strict control over IP addressing. Sometimes cables fail or begin to generate errors: you can check the error rates for an interface with netstat -i.[9]

[9] I have run across these problems most in coaxial-cable (10base2) environments. These sorts of problems are often very hard to track down and are very frustrating. Buy good-quality cable, and label thoroughly.

Sometimes processors will fail. This will almost always induce a panic. However, in some high-end Sun systems (the E3500-E6500 and E10000, for example), the system will automatically reboot, attempt to isolate the fault, and carry on. The result is that a system could seem to spontaneously reboot, but come back up short a few processors (their failure had induced the reboot). It's good practice to use the psrinfo command to confirm that all your processors are online after a reboot, no matter what the cause.

There are many subtle hardware problems that can induce static performance problems. If you suspect that this is occuring, it's usually much easier on your sanity to simply replace the failed hardware. If that's not possible, expect a long and arduous troubleshooting session; it's almost always helpful to have a whiteboard handy to draw out a matrix of configurations to identify which situations work and which don't.


1.4 Concluding Thoughts

Approaching any complex task usually requires some measure of grounding in fundamentals, and a basic understanding of the underlying principles. We've touched upon what performance tuning is, the basic ideas behind computer architecture, the big questions surrounding 64-bit environments, static performance tuning techniques, and some fundamental principles of performance. While it's very difficult to approach the effectiveness of, say, ground school in pilot training, in an introductory chapter, I hope this chapter has given you some sense of the currency in which the rest of this book trades.

I have a suggestion for an exercise, which I think is particularly applicable if you are intent on reading this book cover-to-cover. Go back and reread the five principles of performance tuning (see Section 1.2 earlier in this chapter), put this book down for an hour, and think about what those principles mean and what their implications are. They are very broad, general concepts that are relevant far beyond the context in which they've been presented. Their application is left entirely to you -- in some sense the most difficult part -- although I hope you will find the rest of this book to be an effective guide. When you sit down to analyze a problem, ask yourself a few questions: do I understand what is going on? If not, how can I design meaningful tests to validate my theories? Am I, or my customers, searching for a "free lunch"? What tradeoffs am I making to get the performance I want? Am I overconsuming a resource? Are the metrics I am using to measure performance ones I have developed? Are these metrics measuring what I think they are?

These are all difficult questions, no matter how simple they may appear: they are innocent-looking trapdoors that lead into dark, confusing dungeons. It's not unusual to find eight or ten performance experts in a meeting discussing what exactly is going on in a system, formulating theories, and designing experiments to test those theories. If there is anything you take from this book, I earnestly hope it is those five principles. It's hard to go too far wrong if you use them as your guiding light.


Chapter 2. Workflow Management

The more precisely the position is determined, the less precisely the momentum is known in this instant, and vice versa.

—Werner von Heisenberg, 1927

The topic of "workflow management" is slippery: it can be interpreted to mean many different things. In this chapter, I describe practical means to accomplish the zeroth principle of performance tuning: understanding the environment as it exists. This is the beating heart of dynamic performance tuning. The rest of this book exists simply to improve your understanding about possible environments.

We concern ourselves, as mentioned, primarily with dynamic performance analysis: the system we are measuring is changing beneath us. It is, in some sense, like watching a pond. There might be a creek that flows into the pond; how does that affect the life in the pond? What happens when some beavers build a dam across that creek, when some children find the pond and throw rocks into it, or when someone dumps the ashes of old love letters in the middle of it?

To further complicate things, we are governed by an inviolable principle of physics. Heisenberg's Principle of Uncertainty says that no matter how carefully we try, we will always perturb the system when we measure it, and some piece of knowledge will remain outside of our grasp. We can minimize the perturbation, however, and throughout this chapter we'll concern ourselves with how significant of a perturbation our measurements are inducing.

Things at this point seem pretty daunting. Not only are we measuring something that's always changing in ways we don't understand, driven by factors we can't control, but our mere observation of the system induces further change in it! Why do we bother to measure, then? The answer is simple: without careful, methodical measurements, it becomes practically impossible to provide reasoned, analytical solutions to problems, because we don't understand why the problem exists or what happened. It's always better to know something about the situation. If we didn't strive to understand, we would be doomed to be Cargo Cult Systems Administrators, sitting at our desks with wooden keyboards and terminals,[1] waiting for the magicians to come back in their shiny silver birds at a few hundred dollars an hour to fix our problems for us. That's not a good solution for anyone but the magicians.

[1] Although I have often wished my pager or cellular phone was wooden, and therefore not able to ring.

In this chapter, I break workflow management into two distinct parts: workflow characterization and workflow control. It is important that they are presented in this order; control imposed haphazardly is a real mess, and many a consultant has made a small fortune cleaning up after it. Do not despair about the complexity implicit in this topic. There are many tools that can help us understand and, subsequently, restrict the loads placed on the systems we manage; many of them are minimally intrusive. I then move into benchmarks, as standardized ways of measuring the performance of various workloads on different systems. Consider this your primer on turning Cargo Cult wood into twenty-first century silicon.


2.1 Workflow Characterization

Performance experts are, in some sense, the gurus atop mountains of the computer world.[2] Every time some humble supplicant comes to them with a problem, the guru pokes around at things for a while, then sends the supplicant off to come up with more information. Characterization is the process of trying to gather as much information about the system as possible, so that trends and patterns can be determined. These patterns will prove to be vitally important the first time that performance falls through the floor; we will be able to piece together what the perturbations in the patterns are, and from that figure out what caused them -- rather like studying the wake of a passing ship to see what kind of vessel it was and where it's headed.

[2] To extend this analogy, the database performance people sometimes seem to be in outer space. I have three main points of evidence: it is absurdly expensive to talk to them, either you or they usually have to travel vast distances for an audience, and they speak in strange languages that I don't understand.

An analogy has been drawn between workload management and financial transactions. The first time I read of this was in Adrian Cockcroft's excellent book Sun Performance and Tuning (Prentice Hall). The essential idea is that workload management on computer systems is analogous to a department with a capital budget, staff, etc., performing a task. There are basically three possible outcomes:

  1. If there is no plan and no effective controls on the staff, then the staff will run wild, grabbing as much budget as they can to ensure that their own projects will be well-funded. Some staff will end up with no funding whatsoever, while other staff take up "fact-finding missions" to Maui. The project as a whole ends up a complete mess. (This is known as the "startup model.")

  2. Management is overzealous, and creates a huge bureaucratic staff to plan, assess the plans, and replan. The entire budget is consumed by this bureaucratic middle layer, which micromanages those responsible for actually doing the work, often by demanding daily status reports. The administrative overheads involved make it very difficult to spend any money. The work ends up a mess, because by the time it was supposed to be finished, it's barely been started. (This is known as the "government model.")

  3. In the ideal case, management balances and controls funding so that staff are constrained, but everyone has enough to get their work done. The bureaucratic overhead is kept to a minimum, and status reports are infrequent. The work is done on-time and within its budget. (This is known as the "dreamland model.")

The analogy to performance management is fairly straightforward. Instead of a capital budget, we have the currency of computing, if you will: processor cycles, disk I/O rates, network capacity, etc. We have to have some management over these resources, or we end up following the startup model, where actually getting any resources turns into a grabbing contest. However, if we manage things too much, we'll end up like the governmental model, where nobody can get any work done because we all spend too much time filling out paperwork. So, ideally, we should aim for a balanced approach: the dreamland model. Unfortunately, the dreamland model is aptly named. It's very hard to achieve, much less maintain, such a model in practice.

The first step in performance management has to be developing some processes to guide our decisions. This will involve setting concrete goals, identifying reproducibly measurable performance criteria, and determining the current status. This information then lets us go to management and negotiate for resources to improve the user's experience.

As a discipline, performance management started decades ago in the mainframe world, where everything is stunningly expensive. The payback from making tuning decisions based on real understanding was direct; the actual cost savings in the computing budget was often quite large. Now that the costs of computing have plummeted, it's getting harder to justify the time and effort spent in understanding and tuning. You have to focus more on the indirect advantages to analysis; haphazardly applying system upgrades might not accomplish anything for several months, whereas a careful study of the system and a reasoned upgrade might improve performance significantly. The value of the increased throughput over those several months may be much greater than the cost of the analysis.

There are three important tools we can use to improve our understanding of system behavior: the simple performance measurement commands that we are all fairly familiar with, process accounting, and sar's automated data collection facilities. In this chapter, I'll briefly touch on network pattern analysis.

2.1.1 Simple Commands

We have all used simple commands like iostat, vmstat, and mpstat. These tools are fundamental to performance analysis, and can provide a great deal of valuable information about what is going on in the system. One quick way to gather data about the runtime of a system is to set the interval on these commands to some fairly large amount of time (say, a few minutes), and then redirect the output of the command to a file. One problem with this is that it becomes very hard to keep track of the time that a specific data point was collected. It's straightforward to write some Perl to take care of this for you:

#!/usr/bin/perl

while (<>) { print localtime(  ) . ":  $_"; }

Example 2-1 shows the script in action on a Linux system.

Example 2-1. vmstat on a Linux system
# vmstat 5 | chrononome.pl

Sat Jun 30 00:37:28 2001:  procs               memory      swap      io     system    cpu

Sat Jun 30 00:37:28 2001: r  b  w  swpd  free  buff  cache  si  so   bi   bo  in  cs us sy id

Sat Jun 30 00:37:28 2001: 1  0  0  5472 26680  8420 177908   0  0    10   37  63  37  5  6 14

Sat Jun 30 00:37:33 2001: 0  0  0  5472 26576  8420 177908   0  0     0   20 163  37  1  0 99

Sat Jun 30 00:37:38 2001: 0  1  0  5472 26540  8420 177916   0  0     0   12 148  41  1  1 98

Sat Jun 30 00:37:43 2001: 0  0  0  5472 26904  8420 177920   0  0     2    5 141  43  1  2 98

Sat Jun 30 00:37:48 2001: 0  0  0  5472 26904  8420 177920   0  0     0    1 129  39  2  1 98

Sat Jun 30 00:37:53 2001: 0  0  0  5472 26904  8420 177920   0  0     0    0 124  36  0  1 99

Sat Jun 30 00:37:58 2001: 0  0  0  5472 26904  8420 177920   0  0     0   10 143  35  1  0 99

Some people run these commands with a very short interval via a crontab entry. This is a valid approach, but I am not personally fond of it. It depends on what you want to measure: if I run vmstat with a 1800-second interval, I expect each line to be the average virtual memory activity over the last half hour. However, if I run vmstat with a 2-second interval every half hour out of cron, I get a two-second snapshot of activity right at the half-hour mark. Both can be useful, but I would rather have longer-interval data, at least to start with. If it turns out that I am missing spikes in data, I can always start up another copy of the monitoring application and set the interval to something sufficient to catch the peaks and valleys. Long-term performance trends are not really captured well by increasing the collection interval; mostly you'll just have to do a lot of smoothing on a lot of data.

2.1.2 Process Accounting

Process accounting is a means by which the system gathers information on every process as it runs. This information consists of CPU utilization, disk I/O activity, memory consumption, and other useful tidbits. Perhaps the most useful part of process accounting is that it comes with mechanisms for determining patterns in usage, by means of the audit reporting system.

Some system administrators are scared off of process accounting because of fears of high overhead. Collecting the accounting data has essentially no impact on the system: the kernel always collects the accounting data, so the only extra overhead is the writing of a 40-byte record to the accounting logs. The log summary scripts, however, can take a significant amount of time to run, so they are best scheduled outside of peak hours.

2.1.2.1 Enabling process accounting

Starting system accounting is very simple. In Solaris, be sure that you have installed the optional packages that contain the process accounting functionality: they are SUNWaccu and SUNWaccr. As of Solaris 8, these packages are located on the second Solaris installation CD.

The first step is to link the startup and shutdown scripts:

# ln /etc/init.d/acct /etc/rc0.d/K22acct

# ln /etc/init.d/acct /etc/rc2.d/S22acct

You can then reboot the system, or start accounting immediately by running /etc/init.d/acct start. The second step is to add cron entries to the adm user for the summary reporting commands. Example 2-2 shows what to add.

Example 2-2. Adding cron entries
# min   hour    day     month   wkday   command

0       *       *       *       *       /usr/lib/acct/ckpacct

30      2       *       *       *       /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log

30      9       *       *       5       /usr/lib/acct/monacct

The ckpacct program is used to check the size of the accounting file /usr/adm/pacct. The runacct command generates accounting information from the data, and the monacct generates "fiscal" reports for each user. These reports are stored in /var/adm/acct.

2.1.2.2 Reviewing accounting records

The most useful tool for reviewing accounting data is acctcom, which shows the immediate accounting data. It can be run in several modes: specifying -a will give you average statistics on each process that has been run, -t will provide the system/user time breakdowns for each process, -u user will show all the processes executed by a given user, and -C time will show all the processes that consumed more than time seconds of processor time. There are many other useful options to acctcom that may vary from system to system, so consult your manual pages for more information.

The system accounting programs also generate some files that can be reviewed at your leisure. These are done daily and at the end of every accounting period (defined as when monacct runs).

2.1.3 Automating sar

We will periodically talk about sar as a means of gathering performance data. sar can also be used to automatically collect data and store it for later review.

2.1.3.1 Enabling sar

Enabling automated data collection with sar is quite straightforward, and entails two steps: uncomment the relevant lines (namely, the last 13) in /etc/init.d/perf, and set up the system crontab file to support automated data recording.

Changing the system cron entries (located in /var/spool/crontabs/sys) entails making some decisions about when you'd like data to be recorded (via the sa1 command), and when you'd like data to be reported (by the sa2 command). By default, these are the entries:

# min   hour    day     month   wkday   command

# 0 * * * 0-6 /usr/lib/sa/sa1

# 20,40 8-17 * * 1-5 /usr/lib/sa/sa1

# 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

The first entry will write a sar record every hour on the hour, seven days a week. The second entry will write a sar record twice an hour, at 20 minutes and 40 minutes past the hour, during peak working hours of 8 A.M. to 5 P.M., Monday through Friday.

The third record is more complex. At five minutes past six o'clock, Monday through Friday, it will report on the data gathered between 8 A.M. (-s time) and one minute past six o'clock (-e time), at an interval of 1200 seconds, or 20 minutes (-i seconds), and it will report all data (-A option to sar).

2.1.3.2 Retrieving data

Retrieving data from the sar records is remarkably simple. You simply run sar, specifying the data you'd like to see by means of the normal option switches, and leave off the interval for the present day's data. You can use the -s starting-time and -e ending-time flags to control the time range of interest, and you can specify the day of interest by using -f /var/adm/sa/sa dd (where dd is the day of the month).

2.1.4 Virtual Adrian

Adrian Cockcroft is a distinguished engineer for Sun in the field of performance analysis. He, in coordination with Rich Pettit, developed a software suite for data collection and analysis on Solaris systems. It's described in great detail in their book, Sun Performance and Tuning (Prentice Hall). The book in general is very much worth reading, although it is slightly out of date, and the toolkit is also excellent. This package is called the SE Toolkit, and is located at http://www.setoolkit.com.

2.1.5 Network Pattern Analysis

Historically,[3] networks have not generally been a real-world performance limitation. Consequently, a great deal of effort has been placed on understanding and improving other aspects of the system. Because of the explosion of internetworked applications, the network layer is increasingly becoming a limiting factor. As a result, the tools in network performance analysis tend to not be quite as refined as those in other areas. How, then, can we approach the problem of understanding the network layer in our environment?

[3] That is, in the pre-web world.

A word of warning before we proceed: this section assumes some familiarity with the networking concepts discussed in Chapter 7.

I open this section by discussing three common traffic patterns observed in environments where applications communicate over TCP/IP; then, I briefly discuss some parameters of networked traffic, such as "How large is a packet, on average?" Finally, we'll discuss some of the tools that you can use to determine the patterns occuring on your own network.

A Brief Note on Terminology

I use a few terms in our discussion of network traffic that bear definition:

  • In a TCP/IP connection, the client is the end that opened the connection (the initiating endpoint). The server is the end that accepted the connection (the accepting endpoint). Note that this says absolutely nothing about what roles the machines are playing in anything other than the connection under examination!

  • Inbound traffic is moving from the client towards the server; outbound traffic is the reverse direction.

  • An payload-free packet is generally about 60 bytes on the wire (recall that there are various headers -- typically, Ethernet, IP, and TCP). These packets are a necessary part of how some functions (for example, TCP) work.

  • A small packet is one that is less than 400 bytes on the wire (including all headers).

  • A medium packet is one that is between 400 and 900 bytes.

  • A large packet is between 900 and 1,500 bytes (the maximum for Ethernet).

2.1.5.1 Pattern 1: request-response

The first pattern is that of request-response. The classic real-world examples of this are HTTP, email retrieval protocols such as POP3, and outbound SMTP transactions (delivered to a remote host). Figure 2-1 shows Pattern 1 traffic.

Figure 2-1. Pattern 1 traffic

The request-response pattern is characterized by generally high connection rates. However, fairly low connection rates are sometimes observed, particularly in modern web environments where HTTP session keepalives are being used extensively. The inbound packets are generally small; the outbound packets are generally medium-to-large, but generally there are only a few (less than twenty) of them.

2.1.5.2 Pattern 1B: inverse request-response

A variation on the first pattern is inverse request-response; it is typically seen in inbound SMTP transactions. The interesting part is that the inbound and outbound roles are reversed: the client initiates the majority of the data transfer. Figure 2-2 illustrates the pattern of 1B traffic.

Figure 2-2. Pattern 1B traffic

Connection rates are often moderate, and sometimes exhibit bursty behavior. Inbound packets are generally small in the initial part of the connection, but then become large. The outbound packets are small or payload-free.

2.1.5.3 Pattern 2: data transfer

The second pattern is typical of large amounts of data being transferred. This is most commonly seen in ftp traffic and file transfers during networked backups (see Figure 2-3).

Figure 2-3. Pattern 2 traffic

In general, the connection rate is very low. The inbound packet stream is essentially all payload-free packets, but the outbound packet stream consists almost entirely of full packets.

2.1.5.4 Pattern 3: message passing

The third and final pattern is that of message passing. This is most commonly seen in character-driven applications such as telnet, rlogin, and ssh, but it is also seen sometimes in database transaction schemes such as SQLnet. Some parallelized high-performance computing codes also exhibit this sort of pattern. Figure 2-4 illustrates Pattern 3.

Figure 2-4. Pattern 3 traffic

The connection rate is variable. It is generally low, but can be quite high -- it is very application-dependent. The characteristic to look for is that there are large numbers of small packets being pushed around. This sort of traffic tends to be very hard on a system, because the amount of work required to process a small packet is essentially the same as the amount of work required to process a large packet: you are doing the same amount of work for less payload.

Tuning systems for Pattern 3 workloads is very difficult. If you are faced with a Pattern 3 workload, you are sure the network is the limiting factor in performance, and you need a performance improvement, you have two real choices: either induce algorithmic change in the application so that it stops behaving like Pattern 3 and starts behaving more like Pattern 1, or invest in very low-latency network hardware.[4]

[4] Myrinet has a reputation for fast, low-latency network hardware. It is not widely deployed outside of specialty markets.

2.1.5.5 Packet size distributions

Gathering traces (via snoop or tcpdump, for example) of network activity can give you some very interesting information on what sort of patterns are occuring on your network. One of the more simple questions to ask is "If we draw the graph of packet size versus packet count, what does it look like?" In some of the work I've done on "real-world" systems on the Internet, I found that, remarkably, this graph tends to be trimodal (it has three distinct peaks). These systems are typically web and email servers.

  • The first peak is largely due to inbound traffic, and consists almost entirely of packets about 60 bytes in size. These are TCP acknowledgments flowing "backwards" from browsers towards the web server, acknowledging the receipt of chunks of data.

  • The second peak is almost entirely due to outbound traffic, and occurs at 1540 bytes on the wire. This represents full packets of data flowing away from the server towards the browsers.

  • The third peak took me a little while to figure out: it occurs at about 540 bytes, and is strongly biased towards outbound traffic. It turns out that this is due to certain Windows TCP/IP implementations, which set the "maximum segment size" for a TCP connection to 536 bytes. As a result, the other end cannot transmit any packet larger than 536 bytes back to the client.

If you're in a position where you can strongly influence the traffic on your network, this sort of information can drive great change. Properly tuning application algorithms and TCP stacks can eliminate things like the "middle peak" at 540 bytes, with a corresponding increase in efficiency.


2.2 Workload Control

The next step in the realm of workloads is imposing workload management. This step is often technically straightforward, but it can carry with it political problems. Forcibly attempting to control the users of a system is often not a good idea, and in my opinion should be restricted to cases where education and attempting to secure the cooperation of your user base has failed abjectly. If you are getting some pressure from management to do this, I think you would do well to get it in writing.

Why do I submit these warnings? It is significantly less of a problem in academic environments or in small companies, where everyone knows the systems administrator and it's easy to walk down to their office and ask them what's going on. However, in large professional environments or when the management of the users is divorced to a degree from the management of the administrators, things often get quite complex. It is often very frustrating to users (who, perhaps, designed the hardware or software that you are administrating) to have their abilities restricted: it reduces their flexibility in solving their own problems. If they appeal up through management, you could find yourself trying to explain what seemed like a reasonable technical decision at the time to a few very irate senior vice presidents. You really need to be careful if you are going to be forceful.

On the other hand, user education is one of the most powerful techniques available to you to control workloads. In this section, I discuss user education and the closely related area of written performance agreements, as well as some of the more direct techniques for limiting system resource consumption.

2.2.1 Education

The most powerful tool available to you in order to control the workloads on your systems is user education. Enforcing strict CPU time or disk quotas, while effective, often adds to a "resentment of the mystery" phenomenon. This leaves users feeling rather like medieval serfs: there are certain things they just can't do, like encourage the rain in a dry season, and all they can do is go and beg some rather mysterious people who usually live in caves to try and fix the problem for them. The end result is that the users get very frustrated.

A much better solution is to explain the problem to your users, how their actions induce it, and the solution to the problem. Many times, this sort of forthright discussion will produce the results you'd like, with much less of a headache.

2.2.1.1 Usage and performance agreements

One very basic approach to user education is to develop a usage and performance agreement. This is roughly akin to the "service-level agreements" that are common in the networking world: it is a document that sets out exactly what both sides expect from each other. It can be as formal or as informal as you want it to be. I have seen lengthy documents that looked like they'd gone out for legal review, and half-page descriptions that summarized what the agreed-upon metrics for performance were. In both cases, and in many cases in between, everyone was happy with what they had. I personally prefer short, technically dense documents (I think it's easier to get users to read them), but that's just a matter of taste.

There are many things that can be set out in a usage and performance agreement:

  • What is the minimum acceptable level of performance? That's a pretty loaded question. Ask your users to define it. Are there a certain number of concurrent jobs or users that must be supported? For each job, what is the "maximum runtime"? Can you agree upon a simple test suite that will determine whether, in the general case, performance is "acceptable"?

  • What mechanisms are in place to measure performance? What does the load on the system look like, and what characteristics does it exhibit before failing? Is there a way for users to tell when a machine is "full"?

  • What are the operational windows of the machine? When can maintainance be performed?

  • How are you, the systems administrator, to be notified of problems? If you are out of town or otherwise unavailable, do you have a backup?

  • How are they, the users, to be notified of problems? Is an entry in the message of the day sufficient? Does an email alias need to be created for announcements?

It's vitally important that you revisit these agreements periodically (say, once a quarter). This review may be very short if the document is still accurate, or it may be a bit longer; this is usually the case when the work being done has gotten more complex, or changed to pursue different problems.

It's been my experience that very few people actually take the time to create these sorts of agreements. This is, in my opinion, absurd. In addition to giving you solid information about what the users expect from their environment, it fosters good will and a cooperative spirit. Dealing with uncooperative users is one of the least pleasant aspects of being a systems administrator; unfortunately, it's probably part of your job. Take a few minutes to create an agreement with your users, and get management feedback to ensure your agreement is understood by everybody. It'll be enlightening.

2.2.2 The maxusers and the pt_cnt Parameters

When the Berkeley Standard Distribution (BSD) of Unix was being developed, it created a single tunable that would scale the size of several internal kernel tables and buffers. The scaling factor was directly correlated to the number of time-sharing users that the system had to support, so the parameter was called maxusers. In the modern world, there is no direct relationship between the number of users that a system supports and the value for maxusers. However, a set of parameters in Solaris are still derived from the value of maxusers, and some of them are related to workload control, as described in Table 2-1.

Table 2-1. Parameters derived from maxusers

Variable

Kernel resource

Default setting

max_nprocs

The maximum number of processes systemwide

10 + 16 x maxusers

reserved_procs

The number of processes, systemwide, reserved for the root user

5

maxuprc

The maximum number of processes per non-root user

max_nprocs -- reserved_procs

ndquot

The size of the quota table

(10 x maxusers) + max_nprocs

ncsize

The size of the DNLC(see Section 5.4.1.1)

4 x (max_nprocs + maxusers) + 320

ufs_ninode

The size of the inode cache

(see Section 5.4.2.7 in Chapter 5)

As ncsize

The maxusers parameter itself is automatically sized (unless you are running Solaris 2.3 or earlier). It is set to a value approximately equal to the number of megabytes of RAM in the system; the minimum limit is 8 and the maximum limit (without manual tuning) is 1024, which applies to any system with 1 GB or more of memory. You can set maxusers to a higher value manually in /etc/system, but it is limited to an overall maximum of 2048.

Contrary to popular suspicion, the maxusers parameter does not limit the number of current logins to the system; it just sizes a number of parameters that are generally related to the number of users a system can support. Of course, many things factor into the number of supportable users; maxusers is just intended to be a general knob. The variable that defines supportable users is pt_cnt. By default, this variable is set to 48, which can be a significant limit. There is a practical limit on the value of pt_cnt; namely, the format of the utmp file imposes a limit of 3844 telnet and 3844 rlogin sessions. You should probably keep pt_cnt under about three thousand, in order to avoid hitting other, nontunable system limits. After setting pt_cnt, you will need to create the entries in /dev/pts; this is best accomplished by rebooting with the reconfiguration flag (touch /reconfigure; reboot; this is identical to boot -r).

2.2.3 Limiting Users

Unfortunately, there comes a time when you have to impose limits. Sometimes these limits are set up because user education is very difficult, or because one bad apple can spoil the cider. My personal approach to user limitations is that they are best thought of as an emergency brake; when a (generous) parameter is exceeded, it's healthier to stop whatever is causing the problem rather than cleaning up after it later.

There are two basic approaches to limiting users: quotas and environment controls.

2.2.3.1 Quotas

One of the most common types of user limitations is disk quotas. Disk quotas exist to prevent a user from consuming more than their fair share of disk space, determined by what they're entitled to or what they've paid for.

Disk quotas are typically implemented in two stages. Once a user exceeds a specific threshold of utilization,[5] usually called the hard limit, they are not able to write any more data to disk until they remove some files and their disk utilization drops below that limit. At some lower level of utilization, there is another limit, which is usually called the soft limit. Once the user exceeds the soft limit, they can continue to write data up until they run into the hard limit; however, a timer starts. When the timer (which is usually measured in days) expires, the user cannot write any more files until they remove data and drop below the soft limit. This provides a way for users to temporarily consume large amounts of disk space.

[5] This utilization typically can either be measured in the amount of disk space consumed or in the number of files.

Enabling quotas is quite straightforward on both Solaris and Linux. On Solaris, you will need to install the accounting software packages (SUNWaccr and SUNWaccu) and link two startup files:

# ln /etc/init.d/acct /etc/rc0.d/K22acct

# ln /etc/init.d/acct /etc/rc2.d/S22acct

Then add these lines to adm's crontab file:

# min   hour    day     month   wkday   command

0    *   *    *   *   /usr/lib/acct/ckpacct

30   3   *    *   *   /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log

30   9   *    *   5   /usr/lib/acct/monacct

Then enable process accounting by rebooting or by running /etc/init.d/acct start.

On Linux, the process is similar. The first step is to modify the system startup scripts to turn on process accounting at boot:

# enable process accounting

if [ -x /sbin/accton ]

then

    /sbin/accton /var/log/pacct

    echo "activating process accounting"

fi

You then need to create the accounting record file /var/log/pacct. This file should be owned by root and world-readable:

# touch /var/log/pacct

# chown root /var/log/pacct

# chmod 0644 /var/log/pacct

You can then either reboot or run /sbin/accton /var/log/pacct to turn on process data accounting.

Enterprising users can always come up with ways around disk quotas. I have seen users mail files to themselves, find directories on other filesystems (where quotas weren't being enforced) to hide things in, and even "bounce" files around on the network between multiple machines via some homegrown software. The more reasonable your disk quotas are, the less of a problem you will have with users creatively working around the problem. To return to the emergency brake analogy, think about the electronic speed governors installed on modern automobiles. If the governor is set to something like 140 miles per hour, most people will not ever bother to disable it;[6] they don't ever go that fast, so they may not even know it's there. However, if the speed governor is set to something like 80 miles per hour, a huge number of people will start trying to come up with ways around the problem of their engine cutting off as they try and pass someone on the freeway! The upshot is that the more reasonable you are with disk quotas, the less of a headache they are to enforce.

[6] Just like disk quotas, the enterprising driver can always disable a speed governor.

2.2.3.2 Environmental limits

A more subtle way of controlling users' processes is to impose limits on how much of a system resource their applications can consume. This is implemented at the shell level via the limit command in csh or ulimit in ksh and sh. Table 2-2 summarizes the most common of these parameters and their typical defaults

Table 2-2. Resource limits

Resource

Soft limit

Hard limit

cputime

Unlimited

Unlimited

filesize

Unlimited

Unlimited

datasize

2 GB

2 GB

stacksize

8 KB

2 GB

coredumpsize

Unlimited

Unlimited

descriptors

64

1024

memorysize

Unlimited

Unlimited

Users can increase their own limits up to the hard limit, which can usually be modified by the superuser. Older systems may have different limits imposed by hardware constraints on the data size and stack size parameters (for example, sun4c architecture machines are limited to 512 MB and 256 MB, respectively).

You can place changes to the limits in the default profiles for user's shells. One of the most common changes are to prevent core dumps from occuring by setting the coredumpsize limit to zero:

csh% limit coredumpsize 0

Remember, enterprising users can always find a way around these restrictions.

2.2.4 Complex Environments

In some complex environments, user workload management becomes a very big deal. You may have hundreds, if not thousands, of processors scattered amongst anywhere from one machine to a vast cluster of single-CPU workstations. The problem shifts away from avoiding overloading and toward obtaining optimal utilization across such a wide field of systems. This is a very complex area, and one that commercial vendors are just starting to approach. The leading idea now seems to be computational grids, which enable the distribution of work across geographically disparate areas. While a detailed discussion of these issues is beyond the scope of this text, I would like to offer a few points of advice. Chief amongst these is not to be afraid of gluing together tools into something that works for you. Some Perl wrapped around the process-accounting mechanism can produce nice summary statements of usage on a per-application or per-user basis, with "usage charges" assessed depending on when applications were run and on what machines. You can also, with some care, create simple batch-queuing programs: a user submits their job to a controlling process, which finds an idle system given a certain range of parameters and starts jobs on appropriate systems (e.g., certain machines are workstations and if otherwise idle can only be used between 8 P.M. and 6 A.M., or certain jobs require large amounts of physical memory and should only be run on machines with more than 2 GB of memory). Commercial software exists to perform these sorts of tasks.


2.3 Benchmarking

A benchmark measures the performance of a standard workload. By holding the workload fixed, you can vary the underlying system parameters to generate relative performance numbers.

In general, there are three types of benchmarks: component-specific benchmarks, like SPECint and SPECfp for microprocessors; whole-system benchmarks designed to emulate commercial environments, like the TPC series and SPECweb99, and user-developed benchmarks. In this section, I touch on these different approaches to benchmarking, providing examples of real-world benchmarks throughout.

This book is not about tuning systems to produce world-record benchmark results. In general, producing a world record requires an immense amount of work, strong support from the engineers responsible for the underlying subsystems, and an understanding that the end result is a world record (which may not necessarily be directly applicable to a customer environment).

2.3.1 MIPS and Megaflops

Before we get started in a discussion of benchmarks, we need to discuss a little bit of terminology and history: the MIPS and megaflops metrics.

2.3.1.1 MIPS

One of the first ways of estimating the performance of an application on a computer system was to look at how fast the system could execute instructions. However, this is a dismal metric. The execution rate of a processor (typically expressed in millions of instructions per second, or MIPS) is strongly correlated to the clock rate of the processor. It's probably safe to assume that a 5-MIPS CPU will outperform a 100-MIPS CPU, but a 50-MIPS processor might well actually outperform a 100-MIPS CPU. Using MIPS to compare the performance of different computers is flawed in a basic way: a million instructions on one microprocessor might not accomplish the same work as the same number of instructions on another. For example, let's say you have a processor that does all floating-point work in software, and takes 50 integer instructions to perform 1 floating-point operation; your 50-MIPS CPU now performs 1 million useful instructions per second. Coupled with this is the fact that different instructions can take different lengths of time to complete, especially on CISC processors. Were those million instructions "do-nothing" operations, or floating-point multiplies?

To make things worse, some vendors have redefined the meaning of "MIP" as their own internal metric. DEC was particularly famous for this; for a long time, the VAX 11/780 was considered as the standard "1 MIP" machine. DEC publicly stated that, for a reasonable job mix, the 11/780 was capable of executing about 470,000 instructions per second -- less than half a MIP. For this reason, you sometimes hear people refer to "VAX MIPS" or "VUPs" (VAX units of processing) to make the comparison clear.

The key point to take away from this is that MIPS is only a useful metric to compare processors from the same vendor that support the exact same instruction set with the same compilers. The fundamental weaknesses in MIPS as a performance metric have often led people to joke that it's really short for "Meaningless Indicator of Processor Speed."

2.3.1.2 Megaflops

Floating-point performance is often measured in terms of millions of floating-point instructions per second, often called megaflops or MFLOPS. A floating-point instruction is commonly regarded as one that executes on a microprocessor's specialized floating-point hardware; this includes floating-point additions, multiplications, comparisons, and format conversions. Less commonly included are square roots and divisions. Typically, though, megaflop rates are computed from some mixture of additions and multiplications. While it is considered misleading to present MIPS values, vendors regularly quote peak MFLOPS values. The peak MFLOPS for a processor is usually on the close order of an integer multiple (1, 2, or 4) of the clock rate. This is because optimizations in microprocessor hardware perform fast multiply-add operations (such as for accumulating a running sum of vector elements, like in a dot product of two matrices). As microprocessors become faster and faster, the peak MFLOPS value becomes less useful as a reasonable upper bound on application floating-point performance: memory bandwidth (how fast data can be moved) in and out of the processor becomes limiting.

2.3.2 Component-Specific Benchmarks

The first set of benchmarks I discuss are designed to stress specific areas of the system; typically, they are memory performance and/or microprocessor performance. I discuss two of the most common: the Linpack benchmark and the SPECcpu suite.

2.3.2.1 Linpack

One of the first commonly accepted benchmarks is the Linpack benchmark, written by Jack Dongarra, then of Argonne National Laboratory. Interestingly, the goal of Linpack was not to develop a benchmark; rather, it was intended to provide a high-performance library of routines for linear algebra problems. The Linpack benchmark derives megaflop rates by measuring the time and number of operations required by a few of these routines to solve a dense system of linear equations using Gaussian elimination. There are several versions of Linpack, differentiated by the size of the system of linear equations they worked with, the numerical precision required, and the ground rules for the benchmark. The most typically-quoted was given by the 100 x 100 matrix with double-precision results and strictly compiled FORTRAN (no hand optimizations were permitted).

In addition to the 100 x 100 benchmark, there is also a 1000 x 1000 benchmark, in which the system of equations can be solved via any method of the vendor's choice. The larger Linpack is an important benchmark because it tends to give a reasonable upper bound for the floating-point performance of very large, highly optimized scientific problems on a given system.

The heart of the 100 x 100 Linpack benchmark is a routine called daxpy, which scales a vector by a constant and adds it to another vector:[7]

[7] Daxpy is actually written in FORTRAN; it is represented here in C.

for (i = 0; i <= N, i++) {

     dy[i] = dy[i] + da * dx[i];

}

Each iteration of this loop performs two floating-point operations (a multiplication and an addition) and three memory operations (two loads and a store). It turns out that the Linpack benchmark stressed both the floating-point and memory performance of computer systems up through the early 1990s. When microprocessor cache sizes increased to a size where they could hold the entire benchmark, the Linpack 100 x 100 ceased to be a useful benchmark -- the data structures for the entire benchmark are only 320 KB. The first system that was widely credited with "breaking" the Linpack 100 x 100 was IBM's RS/6000 series. Unfortunately, optimization techniques prevent the Linpack 100 x 100 benchmark from being scaled up to the point where they stress the memory subsystem again.

So, the Linpack 100 x 100 went from being the most prevalent single-number benchmark in history to being relegated to a historical footnote in about two years, as vendors scaled their caches to fit the entire benchmark.

2.3.2.2 SPECint and SPECfp

In the late 1980s, a group then called the Systems Performance Evaluation Cooperative, or SPEC, formed to establish, maintain, and endorse a standardized set of benchmarks.[8] The first SPEC microprocessor benchmark was called SPEC89, and was revised in 1992, 1995, and 2000. By releasing new versions, the SPEC organization keeps current with the latest trends in hardware and compiler design, preventing vendors from becoming too good at tuning their systems to the benchmarks. While the most commonly quoted SPEC benchmarks regard CPU performance, SPEC publishes many standard benchmarks. For more information on SPEC, consult http://www.spec.org.

[8] They are now the Standard Performance Evaluation Corporation.

The first SPEC benchmark was SPEC89, which consisted of the geometric mean of the runtimes for 10 programs. It did not differentiate between floating-point and integer performance, which had the side effect of cloaking performance subtleties. For example, a system that did very poorly on integer performance but very well in floating-point would have the same SPECmark score as a system that performed in exactly the opposite way. Interestingly, when SPEC89 was released, it was not a particularly important benchmark; the different competing architectures of the time had very different performance characteristics, and no benchmark could effectively relate all of their performances. In the field of high-performance computing, the Linpack 100 x 100 code was completely dominant.

To improve the quality of the benchmark, SPEC revised it and released the successor as SPEC92. SPEC92 consisted of 20 test programs, broken up as 6 integer and 14 floating-point codes. The benchmark also reports two summary figures, SPECint92 and SPECfp92, which measure integer and floating-point performance, respectively. As the Linpack 100 x 100 benchmark started to fall, SPECfp92 became the battleground metric for floating-point performance. Around 1994, large caches and smart compilers broke the SPECfp92 benchmark. Thankfully, in 1995, SPEC again released a revision to the benchmark suite. It increased the size of the benchmarks so that they wouldn't fit into caches, greatly lengthened runtime, and became more like large user applications. A similar process occured in 2000, when SPEC again rescaled the benchmark.

The applications under test in the SPECcpu2000 benchmark are quite representative of typical user codes, and are summarized in Table 2-3.

Table 2-3. Codes under test in SPECcpu2000

Class

Code

Language

Description

Integer

164.gzip

C

Data compression

175.vpr

C

FPGA circuit placement and routing

176.gcc

C

C compiler

181.mcf

C

Minimum-cost network flow solver

186.crafty

C

Chess

197.parser

C

Natural language parser

252.eon

C++

Ray tracer

253.perlbmk

C

Perl

254.gap

C

Computational group theory

255.vortex

C

Object-oriented database

256.bzip2

C

Data compression

300.twolf

C

Circuit placement/routing simulator

Floating-point

168.wupwise

F77

Physics (quantum chromodynamics)

171.swim

F77

Shallow water modeling

172.mgrid

F77

Multigrid solver: 3D potential field

173.applu

F77

Partial differential equation solver

177.mesa

C

3D graphics

178.galgel

F90

Computational fluid dynamics

179.art

C

Image recognition

183.equake

C

Seismic wave propagation simulation

187.facerec

F90

Face recognition image processing

188.ammp

C

Computational chemistry

189.lucas

C

Prime number testing

191.fma3d

F90

Finite-element crash simulation

200.sixtrack

F77

Physics (high-energy accelerator design)

301.apsi

F77

Meteorology (pollutant distribution)

The SPEC benchmark reports three numbers for both integer and floating-point performance: the pure metric, the base metric, and the rate metric.

  • The base metric (e.g., SPECint_base95) measures the performance under a restrictive set of rules: applications may only use four compiler flags, the flags must be identical for all applications, etc. The idea is that the base rate will more accurately represent the typically observed performance of user applications.

  • The rate metric (e.g., SPECfp_rate95) measures the throughput of an entire, possibly multiprocessor system. The vendor can run as many concurrent copies of the benchmark as necessary; the rate metric is the aggregate across the entire system.

2.3.3 Commercial Workload Benchmarks

Of course, not all applications are computationally bound; in fact, most business applications stress almost every part of the system. In recognition of this fact, benchmarks have been developed to measure more typical user environments. Here, I discuss two of the most common of these commercial workload benchmarks: the TPC series, which focuses on database performance, and the SPECweb benchmarks for web servers.

2.3.3.1 TPC

The Transaction Processing Council, or TPC (http://www.tpc.org), is an industry-based organization representing computer system and database vendors. TPC provides a framework for measuring the relative performance of these systems. Performance is measured by the amount of time it takes the computer to complete transactions under various loads. The TPC results are extremely well regarded in the database field. Like SPEC, TPC has had to modify and scale their benchmarks over time, as user needs and technological capabilities have changed. Consequently, TPC-A and TPC-B have fallen out of favor; TPC-C and TCP-D are the most common, and TPC-W is gaining favor as an e-commerce benchmark.

There are seven benchmarks in the TPC series:

  • TPC-A was issued in November 1989, and is designed to measure performance in update-intensive, terminal-driven database environments, such as those seen in online transaction processing (OLTP) scenarios. These environments typically have multiple terminal sessions, significant disk I/O, moderate system and application execution time, and transaction integrity requirements. TPC-A is obsolete.

  • TPC-B was issued in August of 1990. Unlike TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B is a database stress test. It tends to emphasize disk I/O without having any remote terminal sessions. TPC-B is obsolete.

  • TPC-C was approved in July 1992. TPC-C is an OLTP benchmark like TPC-A, but is significantly more complex; it implements multiple transaction types and a more complex database. TPC-C simulates a complete computing environment, similar to that of a wholesale supplier, where a population of users executes transactions against a database. The benchmark is based around the principal transactions of an order-entry environment: entering and delivering orders, recording payments, checking order status, and monitoring the level of stock on hand.

  • TPC-D was introduced in December of 1995, and is designed to measure decision-support performance. The test essentially consists of a database that is being slowly modified online while other processes are performing large queries, joins, and scans. TPC-D is obsolete.

  • TPC-H is a decision-support benchmark that consists of a suite of business-oriented ad hoc queries and concurrent data modifications, much like TPC-D. This benchmark was designed to measure the performance of decision-support systems that examine large amounts of data, execute very complex queries, and give answers to critical business questions.

  • TPC-R is another decision-support benchmark. It is very similar to TPC-H, except that it allows additional optimizations based on advance knowledge of the queries.

  • TPC-W is a benchmark designed to measure transactional web e-commerce performance. The workload simulates an Internet commerce environment driven by a transactional web server. There are multiple concurrent browser sessions, dynamic page generation with associated database accesses and updates, consistent web objects, online transaction execution modes, transaction integrity, and contention on data access and update.

2.3.3.2 SPECweb99

As the Web began to explode in popularity in the mid-1990s, SPEC released a benchmark for web performance, called SPECweb96. SPECweb96 was focused entirely on the speed at which static pages could be retrieved by a set of clients. SPECweb96 was eventually broken by a software technique pioneered by Sun Microsystems, called the Network Cache Accelerator, or NCA. NCA uses a kernel module to transparently cache static web content in a kernel memory buffer, and replies to HTTP document requests for documents in its cache without ever waking up the application web server.

By the end of the 1990s, there was an increasing trend toward dynamic content -- the precise contents of a page depended on form data or was in some other way personalized to the end user. Driven by this change, SPEC released a new revision to the SPECweb benchmarks, which is known as SPECweb99.

SPECweb99 measures the number of concurrent conforming connections per second; in order for a connection to conform, it needs to sustain between 400 and 320 Kbps. The transaction mix includes both static and dynamic content; the default mix is shown in Table 2-4.

Table 2-4. SPECweb99 transaction mixture

Request type

Percentage

Static GET

70.00%

Dynamic GET (plain)

12.45%

Dynamic GET (custom advertisement)

12.60%

Dynamic POST

4.80%

Dynamic POST calling CGI code

0.15%

The static portion of the workload models a hypothetical web content provider. The pages of content are represented by files of different sizes that are accessed with varying frequency; SPEC obtained the precise values of these parameters by inspecting the log files of a handful of busy web sites. The working set of content files fit into four classes, as illustrated in Table 2-5.

Table 2-5. Static content classes

Class

Size range

Size step

Percentage of hits

0

< 1 KB

0.1 KB

35%

1

< 10 KB

1 KB

50%

2

< 100 KB

10 KB

14%

3

< 1000 KB

100 KB

1%

The workload file set consists of a number of directories, each of which contains 9 files per class, for a total of 36 files per directory. The files in Class 0 are in increments of 0.1 KB, those in Class 1 are in increments of 1 KB, etc. For example, the files for Class 2 will be 10 KB, 20 KB, 30 KB, 40 KB, 50 KB, 60 KB, 70 KB, 80 KB, and 90 KB. Since larger web servers are expected to service more files, the exact size of the workload file set (the number of directories to be created) is determined by the number of simultaneous connections.

During a benchmark test, the class of file to be retrieved from a random directory is determined by a fixed distribution (illustrated in Table 2-4). A Zipf distribution is used to choose which file should be retrieved from the selected class.[9] The relative hit rates for each file in a class are shown in Table 2-6.

[9] The probability of selecting the nth item in a Zipf distribution is proportional to 1/n. Zipf distributions are usually associated with situations where there are many equal-cost alternatives, such as picking books at a public library.

Table 2-6. Relative hit rates of files in a given class

File number

Percentage

0

3.9%

1

5.9%

2

8.8%

3

17.7%

4

35.3%

5

11.8%

6

7.1%

7

5.0%

8

4.4%

SPECweb99 is an emerging benchmark, and vendors are starting to play a lot of tricks to improve performance that you might not want to play on a production system. If you're trying to use SPECweb99 to evaluate systems for purchase, please take them with a grain of salt, and read the reporting configuration very carefully; vendors are required to report exactly what hardware was used and what parameters were tuned. [10]

[10] Note that the way you tune a system when you're going for a world record can be quite different than the way you'd tune a system that you had to use every day in production. Vendors take world records very seriously.

2.3.4 User Benchmarks

The most meaningful benchmarks are the ones you develop that are relevant to your environment. In general, there are a few principles that you should follow to set up a good benchmark: they should be relevant to your typical problems, have a relatively short runtime, be extremely automated, and set rules for running the benchmark.

2.3.4.1 Choose your problem set

In general, pick problems that represent the spectrum of work you actually do. If, for example, you are buying a computer for genetic sequence analysis, you might want to pick a few representative sequences, each with a separate computational code. It's okay if most of your work uses one particular application -- just weight the performance of that application higher. You should also design the benchmark so that it tests not only the work that you are doing now, but the work that you plan on doing over the lifetime of the machine (say, three to five years). Try and benchmark the big memory, high computation problems, or a reasonable subset thereof, that you'll be doing five years from now.

2.3.4.2 Choose your runtime

Of course, it's hard to predict runtime. One of the best guidelines for this is a well respected, industry-standard benchmark like SPECfp2000. If your current system is rated at 100 SPECfp2000, a system that's rated at 1000 SPECfp2000 should go ten times as fast (work with me, here). A benchmark that took 20 minutes to run on your current system would be reduced to a trivial 2-minute run on the new system. As a rule of thumb, each "problem" should run for 10 to 20 minutes on the new hardware under consideration. It should take less than an hour or two to run the entire suite, if for no other reason than you will have to sit down and run it many, many times.

2.3.4.3 Automate heavily

You, or a vendor trying to get you to give them money, will be running this benchmark frequently. As a consequence, you will want things to be as automated as possible. The best way to do this is to write a wrapper (often a shell script) that will do everything from compiling the benchmark, to running it, to printing a summary of the results, along with a snapshot of the most important elements of the system configuration. These elements might include things like the number of configured processors, the amount of memory installed, or the compilation flags used.

2.3.4.4 Set benchmark runtime rules

It is important to set rules, especially if anyone other than you will be running the benchmark. There are a lot of questions to keep in mind when writing benchmark rules: are modifications allowed? Are code changes only allowed for portability, or can algorithms be replaced? What type of compiler optimizations are allowed? How closely must the benchmark results match your baseline results? In fact, you probably need some rules about the rules, just to make sure you have a clear understanding with the vendor.

We've only touched on the highlights of benchmark design. I strongly recommend that you refer to Kevin Dowd and Charles Severance's High Performance Computing (O'Reilly), for a much more in depth discussion of this topic.


2.4 Concluding Thoughts

I hope this chapter has provided some theoretical and practical insight into understanding and managing workflows. I've discussed means of understanding the workload impressed upon a system, as well as some means of enforcing limits upon that workload. I have also talked about benchmarks as means of comparing the performance of systems under standardized workloads.

Parts of this chapter may seem almost childlike in their simplicity. As we progress into the performance analysis of dynamic systems, we are presented with simple things that are actually complex. Part of this book's purpose is to examine some of the simplifications, so that the reasons behind become apparent. This exploration lets you make better decisions about where to direct your tuning efforts.


Chapter 3. Processors

Quote performance in terms of processor utilization, parallel speed-ups, or MFLOPS per dollar.

—D. H. Bailey, "Twelve ways to fool the masses when giving performance results on parallel supercomputers." Supercomputing Review, 1991

The increase in microprocessor performance over the last 20 years has been phenomenal. The prediction made by Gordon Moore, the founder of Intel, that transistor counts on microprocessors would double every 18 months (and thus performance would increase proportionately), has certainly held true, if not been exceeded. In the early 1980s, the fastest microprocessor from Intel was the 4004, which was clocked at less than 4 KHz; by comparison, in mid-2001, the latest generations of IA-32 processors from Intel and AMD are running at clock rates of about 2.0 GHz -- and execute more than one instruction per clock cycle (I survey some the key elements in microprocessor design later in this chapter). This jump is so huge that it is hard to comprehend. If automobile performance followed the same growth curve as microprocessors over the last 20 years, the successor to the DeLorean DMC-12 (1981-1983), which had a top speed of perhaps 140 miles per hour,[1] would have doubled 12 times: it would travel at almost 287 thousand miles per hour, which is equivalent to about 410 times the speed of sound, or about one-sixteenth of the speed of light. Even that's pretty difficult to comprehend. If a pack of bubble gum cost a quarter in 1983, and prices increased at the same rate as microprocessor performance, that same pack of gum would cost just over a thousand dollars.

[1] The DeLorean DMC-12, despite being positioned to compete against the Corvette as an American sports car, was quite heavy, due to its extensive use of dairy-grade stainless steel.

It's not clear whether this rate of increase is sustainable. It has often been predicted that the rate will have to slow. However, enterprising engineers have found ways to improve the technologies used for processor fabrication and design so that this has not yet become the case.

The fact that microprocessor performance increases have far outpaced the performance increases of the other components in a computer system is a unifying theme for this book: how do we make the other components go as fast as possible, to try and optimize the efficiency of the processor? Processor performance is without a doubt the most-quoted figure in computer performance. This can easily be verified simply by going to the local computer store and looking at their advertisements; the information in the largest print, often set apart and highlighted, is the clock rate of the microprocessor. It is much more appealing to have the latest 2.0-GHz processor than a few very fast hard drives. However, while it is absolutely true that microprocessor performance is a critical component of overall system performance, it is often overrated.

Despite the central importance of the microprocessor, to most system administrators it is little more than a replaceable black ceramic box with a heat sink attached. While it's true that most systems administration tasks don't require any significant knowledge of electrical or computer engineering, performance tuning is at heart about understanding how things work -- it's very difficult to improve something if you don't have any idea how it works. For processor performance, this foundation is microprocessor design. In this chapter, I move from discussing basic microprocessor architecture to a host of supporting areas: caches, which play a vital role in the performance of modern processors; process scheduling, or how the operating system decides what processes should have priority in using the CPU; multiprocessing; the interconnects used to connect processors to other processors and to peripheral devices; and, finally, I discuss some tools that can be used to monitor microprocessor performance.


3.1 Microprocessor Architecture

At a fundamental level, a microprocessor is the physical implementation of a set of rules. These rules, called instructions, specify exactly what tasks the microprocessor is allowed to perform. The set of all the rules together is called the instruction set; in combination with other information, they define the instruction set architecture, or ISA, which contains all the information you'd need to write a program that runs correctly. For a more detailed discussion of these abstract underpinnings, see Chapter 1.

In the last 20 years, microprocessor designers have split into two competing camps based on design philosophies. One camp espoused creating an instruction set in which individual instructions were very powerful, almost on the level of C primitives. These instructions are quite complex, and this camp came to be known as building complex instruction set computing (CISC) designs. The other camp decided to create instruction sets in which the individual instructions were minimal, but their simplicity allowed many optimizations to be performed. This is known as reduced instruction set computing (RISC) processor design. Both methods accomplish the same amount of work. The difference is that a RISC design will probably take more instructions, but those instructions will most likely execute in less time.

RISC has won the war of performance. Increasingly, CISC processor designs are becoming more RISC-like. For example, Intel's P6 processor core actually translates the complex IA-32 instructions into a much simpler internal format before executing them. Another good example is the Transmeta Crusoe processor, which uses specialized hardware (called code morphing) to, in part, translate IA-32 instructions into the Crusoe's internal instructions.

Lately, designers have started to add extra instructions, typically for graphics support. Sun's VIS and Intel's MMX extensions are good examples of this.[2] These instructions are based around the idea that while the smallest addressable piece of memory is typically 32 bits long, we might only operate on 8 of those bits, so we perform 4 memory operations and 4 arithmetic operations when the data would have fit in 1 memory operation. So, these extensions work by packing the full 32 bits of data in, so that only 1 memory operation and only 1 arithmetic operation is required. However, using these extensions to maximum effect involves quite a bit of effort on the part of the compiler design team. Figure 3-1 shows multimedia extension parallelism.

[2] Intel's IA-32 architecture is, as we've said, a CISC design. MMX is still a good example of a graphics-specialized instruction set, however.

Figure 3-1. Multimedia extension parallelism

So, are RISC processors with these extensions added on (for example, Sun's UltraSPARC) still RISC? The important thing to remember is that RISC was a means to an end -- building the fastest microprocessor possible within specific technological constraints -- not an end in and of itself. Some people have taken to saying that these "second generation" RISC processors aren't actually RISC at all: instead they implement fast instruction set computing (FISC), which perhaps defines their intent a little more clearly.

3.1.1 Clock Rates

Let's imagine our microprocessor as an Athenian trireme[3] moving through the Mediterranean. In order for the ship to move forward at peak efficiency, all the rowers must row at the same pace: their actions are synchronized by the beat of a large drum. In much the same way, everything on a microprocessor happens in synchronization by means of an external clock signal. This signal, which is typically cycling millions of times per second (one hertz is one cycle per second; processor clock signals are generally counted in megahertz (MHz), or millions of hertz), is usually generated by a clock crystal on the processor's logic board. In large datacenter systems, the clock signal is sometimes generated on a dedicated clock board. This clock rate is increased by a set multiplier to produce the internal clock rate, which is the number that is typically counted. This multiplier determines how fast a given processor will perform in comparison to other similar processors.

[3] A trireme is a seafaring vessel, typically a warship, characterized by three rows of oars. The Greek city-state of Athens was known for its naval strength, much as Sparta was known for its army.

I can't emphasize enough how dangerous it is to make performance comparisons based only on clock rate. Table 3-1 is a comparison of the IBM PowerPC 604e microprocessor, which is used in their RS/6000 line to the MIPS R10000, which is used by SGI.

Table 3-1. Processor/clock rate performance comparison

Processor

Clock rate

SPECint95

SPECfp95

IBM PowerPC 604e

332 MHz

12.9

6.2

IBM PowerPC 604e

266 MHz

9.2

5.8

MIPS R10000

250 MHz

12.4

9.7

We'll talk more about the SPEC benchmark suite later (also see Section 2.3.2.2 in Chapter 2), but a gross analysis of the data is possible here. From a simple comparison of the clock rates, it would seem that the MIPS R10000 processor, at a mere 250 MHz, should be slightly slower than the 266 MHz PowerPC, and 25% slower than the 332 MHz part. However, the benchmarks tell a very different tale: the R10000 is almost as fast as the 332 MHz PowerPC in integer performance, and bests it by a substantial margin in floating-point computation.

The astute reader will have noticed that Table 3-1 also provides evidence that increasing clock rate does not imply an equivalent increase in performance. While the 332 MHz IBM PowerPC processor has an almost 25% higher clock rate than the 266 MHz version, it only has a slight (6.8%) edge in floating-point performance. This is because increasing the clock rate of the processor tends to accentuate the constant and substantial delay involved in accessing memory. This is particularly an effect of floating-point benchmarks, because they tend to be rather large.

The overall limit of the clock signal is dependent on some rather esoteric parameters in the processor design, such as the underlying semiconductor technology. While it is possible to optimize for clock speed by adjusting these parameters, the real-world characteristics -- such as cost, heat, and power consumption -- of the resulting design may make it infeasible for the marketplace. However, the clock rate also depends on the complexity of work being done. It takes a certain amount of time for the processor to execute a specific task. The slowest path to take through the processor is called the critical path. Reducing the critical path, such as by simplifying the design, lets the clock speed go ever higher.

Another way of thinking about the performance of a processor is to consider how many cycles it takes to execute an instruction, on average. This is called the cycles per instruction (CPI); obviously, lower numbers are better.

3.1.2 Pipelining

Since we can't always increase the performance of a processor simply by increasing the clock rate, we must be more devious in our approach. One method involves placing two functional units in the design; for example, instead of having just one unit that can perform integer arithmetic, we'll have two.[4] However, this is rather expensive in terms of processor die space.

[4] A processor that uses this technique is called superscalar. I discuss this in Section 3.1.3 later in this chapter.

Instead, we might look closer at how a microprocessor operates. While an instruction is being retrieved from memory, the unit responsible for integer arithmetic is sitting idle; surely this isn't the best way to make a fast processor! For example, we might design a processor that breaks up instructions into five stages:

Instruction fetch (IF)

Retrieve the next instruction to be executed from memory.

Instruction decode (ID)

Figure out what the instruction actually does.

Execute (EX)

Execute the instruction.

Memory (MEM)

Write data to and read data from memory.

Writeback (WB)

Write computed results into registers.

A single instruction might not use all of these stages -- for example, a memory load instruction might not have any need of the circuitry provided in the execute stage.[5] So, let's try overlapping the execution of the instructions.

[5] Then again, it might. This depends on the particulars of the processor design.

The first instruction is initially fetched. After it moves onto the decode stage, the next instruction is fetched. As the processor continues to work, it occupies all of itself in this fashion, so that in a given instant there are five instructions being executed at once. This process is called pipelining.

In a five-stage pipeline such as we've described, we ideally get the results of an instruction every clock cycle. However, this does not mean that each instruction takes only one cycle to execute; rather, each one takes five cycles. It's important to be careful about this when you're thinking about CPI measurements; in a pipelined processor operating perfectly, the CPI is 1.0. This just means that, on average, one instruction is completed every clock cycle. So, pipelining does not increase the performance of a single instruction; rather, it increases the throughput of the microprocessor. In addition, as we break each instruction up into a set of stages, we reduce the complexity of each stage. The more stages we have, the simpler we can make each one, which means that we can probably increase the clock rate of the processor as well. Figure 3-2 illustrates instructions in flight through a pipeline.

Figure 3-2. Instructions in flight through a pipeline

Pipelining is one of the most dominant factors in the historical improvement of processor performance. However, it is not quite so simple as we have made it out to be. Sometimes, an instruction might take a particularly long time; for example, reading data from memory might take substantially longer if the data to be read doesn't reside in a cache. This causes the pipeline to "gum up," which is called a stall. Microprocessor designers go to great lengths to avoid pipeline stalls; stalls have a very detrimental effect on performance.

3.1.2.1 Variable-length instructions

The smooth functioning of our pipeline is dependent on all our instructions taking approximately the same amount of time. In a RISC design, each instruction is generally the same amount of data. However, because of their complexity, CISC instructions often vary in length. For example, a "return from routine" instruction is pretty compact, but a complex "multiply-add" instruction might be a lot longer. Figure 3-3 illustrates the difference between fixed-length instructions and variable-length instructions.

Figure 3-3. Fixed-length and variable-length instructions

The processor has no idea how long an instruction will be until it decodes it and finds out what it is. Since the processor has to fetch the instruction before it can be decoded, if the instruction is longer than what was fetched, the processor will have to go back to the instruction fetch stage and recover the rest. This involves a costly trip to memory, and stalls the pipeline. If, as in most RISC instruction sets, the length of each instruction is fixed by definition, then this problem doesn't occur, and the pipeline can flow more smoothly.

3.1.2.2 Branches

Another problem in pipelining is instructions known as conditional branches. A conditional branch specifies the next instruction to be executed, depending on a specific computation. Consider the following code snippet:

if (i == j) {

   delta++;

}

The if statement here is an example of a conditional branch. These are implemented in most RISC instruction sets as branch if equal (BEQ) and branch if not equal (BNE) instructions.

When the processor retrieves an instruction from memory, it has no idea what the instruction is. So, it fetches the next instruction in memory. By the time the processor knows that the instruction is a branch, it has finished decoding it. If the branch falls through (that is, the next instruction is the direction the branch goes), everything works well. However, if it turns out that the branch is taken (that is, the next instruction executed is not the instruction that was fetched), the processor needs to go back and remove all the pending instructions in the pipeline and fetch the next instruction we want to execute.

Unfortunately, analysis has shown that branch instructions occur on average about every seven instructions. If only half of our branches fall through, we lose about 20% of the pipeline's efficiency. Rather than take a penalty for clearing out the pipeline, many RISC processors implement a branch delay slot. This means that the instruction that goes into the pipeline right after the branch is either harmless or useful, whichever way the branch goes. For example, consider the pseudocode below:

loop:

   c--;

   i = j++;

   j = k + 4;

   l = m + n;

if (c == 0) goto loop;

Because adding m and n and storing the result in l does not depend on any earlier instructions in the loop, we can reorganize the instructions like this in order to use a branch delay slot:

loop:

   c--;

   i = j++;

   j = k + 4;

if (c == 0) goto loop;

l = m + n;

By the time the processor has decided which way the branch should go, that instruction is about to be fetched. If we can't find a good instruction to fill the branch delay slot, we simply insert a no-operation (NOP) instruction. This is generally handled by the assembler and is not dependent on the programmer. This is a neat way to reduce the impact of branch operation-related processor stalls; however, as processors became capable of running more concurrent instructions, more powerful approaches were needed.

If we could decrease the branch miss rate further, the pipeline would be more efficient. With this in mind, processor designers devised schemes that allow the processor to predict the direction of a branch. As an instruction is decoded, the processor notices that it is a branch and consults a table of the recent operation of the branch. Based on this recent behaviour, it makes a guess and immediately begins fetching instructions from the guessed location. If the prediction is wrong, we have to cancel all our current instructions. One trivially simple mechanism of branch prediction is to always assume that the branch that refers to an earlier address is always taken. This is surprisingly accurate, because many programs iterate over a loop many times:

for (i = 0; i <= 100000; i++) {

   j[i] = k[i] + j[i];

}

If the branch prediction is correct, branches are no more expensive than any other instruction. If not, we must stall the processor, but hopefully this won't occur very frequently. A simple branch-prediction scheme is over 95% correct, which significantly reduces the overall penalty of branch-related pipeline stalls.

3.1.3 The Second Generation of RISC Processor Design

The ideal first-generation RISC processor would be able to execute one instruction per clock cycle. Although a five-stage pipeline would take five instructions to fill completely, if we kept it full, the overall CPI would start to grow close to 1.0. Once this was approached in advanced first-generation designs, microprocessor design companies began to compete to see who could build the fastest RISC processor. These strategies fell into three basic areas: speed freaks, superscalar designs, and superpipelined designs.

Speed freak designs are based on optimizing a simple, small, efficient core for very high clock rates. This approach is epitomized by the Digital Equipment Corporation (DEC) Alpha processor, which has historically exhibited clock rates approximately twice that of the nearest competitor.

By adding more computational elements to the processor die, performance can be increased without increasing the clock rate. This is known as superscalar processor design. The number and type of operations that can be run concurrently is dependent on both the application, which must have some degree of parallelism, and the processor, which must have enough functional units and a way of keeping them busy. This idea seems simple, but it is quite difficult, on the part of both the compiler writer and the hardware engineer. Current state-of-the-art superscalar processors can issue six instructions per clock cycle. At peak performance, they achieve a CPI of 0.17!

One variation on superscalar processing is called very long instruction word (VLIW) design. In such a scheme, the processor itself is very simple, and lacks almost all of the advanced scheduling hardware found in superscalar processors. In a VLIW processor, such as the Transmeta Crusoe, instructions are grouped together into bundles (instruction words) of three or four.[6] The processor then loads each instruction word in sequence and executes all of the instructions in parallel on multiple functional units. This causes an incredible burden to be placed on the shoulders of the compiler writers: writing an optimizing compiler for a VLIW processor is exceptionally hard, and is still an area of active research. The Crusoe design lightens this burden by implementing what amounts to an on-the-fly recompiler in hardware.

[6] In the early VLIW processors built by Multiflow Computer, as many as 28 operations could be bundled in a single instruction word.

In RISC processors, simpler often means faster; perhaps some of the instruction steps in your pipeline are overly complex. The hallmark of a superpipelined design is to reduce per-step complexity by increasing the number of steps. If the reduced complexity allows the processor to execute faster, superpipelined processors can achieve the same performance improvements as superscalar processors. However, deep pipelines tend to mean that branches cannot be resolved until relatively late in the pipeline, which makes mispredicted branch stalls very expensive. Superpipelining can also be combined with other approaches, as is evidenced by the MIPS R8000, which was a superscalar design with a pair of deep pipelines.

Designing a fast computer is more than just a function of a fast microprocessor, however. Removing the bottlenecks in processing exposes more bottlenecks, particularly in the way that information is moved in and out of the CPU.


3.2 Caching

A major concern of our microprocessor optimization strategy is that pipeline stalls cause serious performance problems. If every memory request has to be satisfied from main memory, the pipeline will stall very frequently. Modern microprocessors universally implement caches to try and solve this problem. In some processors, this cache is organized as a single unit, called a unified cache architecture. More commonly, however, the cache is divided up into two areas: one for data and one for instructions. This is called a Harvard cache architecture. Harvard caches tend to be significantly more efficient in the real world, especially for small caches.

3.2.1 The Cache Hierarchy

Memory speed has increased at a much slower rate than processor speed: processor performance doubles roughly every eighteen months, but memory performance doubles roughly every seven years. The important thing to realize is that as processor speed has outpaced memory speed, even static RAM caches located off the processor die are not fast enough.[7] The total time required to move data to or from memory is more than just the memory's physical access time, as there is overhead in moving data between the chips, as well as keeping the caches synchronized with main memory. The most common mechanism for minimizing access time is to create a hierarchy of caches with steadily increasing size and steadily decreasing performance:

[7] We'll talk about the various types of memory in greater detail in Chapter 4.

  • The processor has separate caches for instructions and data; they are referred to as the i-cache and the d-cache, respectively. These are often about 16 KB each, and the fastest caches in the system. They are called level 1 or L1 caches.

  • There is a larger, combined data/instruction second-level cache, which is generally between 256 KB and 4 MB in size. A cache miss at the L1 cache can still produce a cache hit in this cache -- while it takes a few cycles longer it is still very efficient. This is the level 2 or L2 cache.

  • Sometimes a third-level cache is configured. It is usually 4 to 32 MB, and while it is still slower than the L2 cache, it is roughly an order of magnitude faster than main memory.

  • If cache misses occur at all these levels, main memory must be referenced. If the memory system is very loaded, there could be a long delay. We want to avoid that.

Most extremely high clock rate microprocessors have developed a three-level cache structure. An example of cache levels and their access times for a typical 1.0 GHz processor can be found in Table 3-2.

Table 3-2. Typical memory access speeds

Cache location and type

Access time (in ns)

Size

"Good" hit rate[8]

L1 on-chip

1-3

8-64 KB

>90%

L2 on-chip

6-18

1-8 MB

>50%

L3 off-chip

30-40

8-32 MB

>30%

Main memory

220

Very large

-

[8] This is best described as "the percentage of requests to the cache that can be fulfilled with data already in the cache."

Every time data is loaded from any lower cache level, it's loaded into the caches above it as well. When a reference to memory is located in a cache, we say we have a cache hit; when we have to "fall through" the cache and go look in the next level of the hierarchy, we have incurred a cache miss. Every cache miss incurs a miss cost. Predicting the miss cost is often complicated: newer processor designs like Sun's UltraSPARC have variable cache miss costs. In general, the miss cost is proportional to the difference in speed between the processor clock rate, the cache clock rate, and the main memory speed. It is very good from a performance viewpoint to hit the cache as often as possible. Let's look at the average memory-access time of our prototype system based on two distinct scenarios:

  • We hit the L1 cache 65% of the time, the L2 cache 25% of the time, and main memory 10% of the time.

    0.65(4ns)+0.25(5ns)+0.10(220ns) = 25.85ns

  • We hit the L1 cache 90% of the time, the L2 cache 8% of the time, and main memory 2% of the time.

    0.90(4ns)+0.08(5ns)+0.02(220ns) = 8.4ns

While 8.4 nanoseconds sounds awfully good, it's important to keep in mind that the clock cycle time of this system is on the order of a nanosecond. Nonetheless, as should be apparent, it is important for the cache hits to happen as much as possible and as high up in the hierarchy as fast as possible (Figure 3-4).

Figure 3-4. Performance as a function of L1 cache hit rate

3.2.2 Cache Organization and Operation

Caches are organized into equal-sized chunks called lines. A line typically contains four to sixteen sequential memory locations: although the data in a line is physically contiguous in main memory, two distinct cache lines can contain data that is quite separated in memory.

When data is requested from the cache, the system checks to see if the data is in one of the cache lines. If it is, then it is returned quickly; if not, the request falls through the next cache level (or to main memory). When the data is retrieved from the lower cache level, it is pulled into a line in the caches, discarding an existing line if necessary.

It is important to note that the data in a cache is not the "master copy" of the data. So, when data is modified in a cache, systems administrators have three choices:

  • Update the copy in the cache only, which is very fast. However, now the master copy and the cached copy differ. This is write-back behavior.

  • Update both copies immediately, which may be slow. This is efficient if the cost of writing to the cache is low. This is called write-through.

  • Throw away the cached copy and just update the master. This is only a good idea if the cost of writing to the cache is high.

Not all of main memory is cacheable: typically regions that involve I/O devices are uncacheable. This is determined by the address range, not by the data or instructions stored in the memory location.

3.2.3 Associativity

Every cache line is paired with the canonical data stored in main memory in a process called mapping. The cache maintains a record for each line, called a tag, of what memory addresses the cache line is associated with; the last access time is also stored in some designs. These maps can be constructed in three different ways:

Direct mapping

This is the most straightforward way of mapping main memory to cache lines. In this scheme, cache line zero stores integer multiples of the cache size. For a 32 KB cache, the memory addresses that were 0 modulo 32 KB (0, 32 KB, 64 KB, etc.) would be mapped against the first cache line. This is a cheap and simple method to implement, but it can cause decreased performance (called thrashing) when working with two memory areas that are an exact integer multiple of the cache size apart. For example, when copying the contents of an array A into an array B, and A and B are exactly 32 KB apart, A will be read, then B has to be pulled into the same cache line to be written; this incurs a lot of cache misses very quickly.

Fully associative mapping

Fully associative mapping allows any cache line to be mapped against any memory address. Unlike a direct mapping implementation, each cache line in a fully associative cache knows something about the data it contains. When the processor requests data, each of the cache lines is queried as to whether it contains the data. If none of them have it, then a cache miss occurs. The cache lines track when the last access time of the data occurred, so that when a line needs to be overwritten, the oldest data can be discarded. Fully associative caches tend to have high utilization percentages, but are very expensive.

Set associative caching

A set associative cache is a compromise between the direct-mapping and fully-associative methods. They consist of a grouping of two or four banks of direct-mapped cache. Every time a cache line needs to be replaced, there is more than one choice of where it can go. This prevents the cache thrashing problems that are observed in direct-mapped caching.

3.2.4 Locality and "Cache-Busters"

Caches are so successful at what they do because programs exhibit peculiar memory access characteristics, which are called temporal locality and spatial locality. This means that, after an instruction or data access, the next instruction executed or the next data bit fetched tends to be close in either space or time. Temporal locality depends on reusing something repeatedly; spatial locality depends on using things at the same time that are close to each other.

We call a piece of code that causes the cache functionality to fail a cache-buster. Typically, a cache-buster breaks the temporal/spatial locality that the cache relies on for efficient operation. Three of the most common are non-unit stride, linked lists, and cache-aligned block copies.

3.2.4.1 Unit stride

Caches work best when a program is working through memory sequentially. Let's say that we are using short (16-bit) integers, and our cache lines are 128 bits in length. When we reference the first integer, the next seven will be loaded into the cache line as well, and our next seven references will be satisfied very quickly. This is called unit stride, because the address of each successive data reference is one greater than the data reference previous. The following code exhibits unit stride:

for (i = 0; i <= 100000; i++) {

   j = j + Array[i];

}

If a program exhibits non-unit stride, its performance suffers because each memory reference requires a trip to main memory. This is an example of a non-unit stride loop:

for (i = 0; i <= 100000; i += 16) {

   j = j + Array[i];

}

Even though this code performs one-sixteenth of the work of the unit stride loop, the two loops perform identically! Noncached memory accesses dominate the performance of the non-unit stride loop. For more examples of loop optimization, see Section 8.3 in Chapter 8.

3.2.4.2 Linked lists

Let's think about a linked list with a few thousand entries. Each entry consists of some data and a link to the next element. We search our list by traversal: because the code for searching the list can be made very compact, it fits well in the instruction cache. However, every data request will be in a different cache line, and so every data access incurs a cache miss. If the size of the linked list exceeds our cache size and we are forced to start flushing cache lines to make room for the next element in the list, our next search attempt will not find the start of the list in the cache! The key point is that list elements are rarely congruent in memory; therefore, iterating over them breaks the spatial locality principle that our caches depend on.

The only solution to this problem is to devise a new algorithm. As processors get faster, the problem will only get worse, since the processor/cache clock rate and the memory access time disparity will only grow larger.

3.2.4.3 Cache-aligned block copy problems

The kernel spends much of its time copying and zeroing blocks of data, particularly in page-sized (typically 4 or 8 KB) chunks.[9] This data does not need to be cached, since it probably won't be used again immediately; in fact, it will simply serve to remove useful data from the cache. The most efficient way to copy a page-sized piece of data with write-back caching is to read a cache line, then write it as a block, as shown in the following steps:

[9] I discuss pages in Chapter 4.

  1. Read the first piece of the source, incurring a cache miss.

  2. Read the entire cache line of the source.

  3. Write the first piece to the target, incurring a cache miss.

  4. Write the entire cache line to the target.

  5. Repeat until finished. At some later stage, the target cache line will be flushed for more useful data.

If the target and source are separated by an exact multiple of the cache size, and the system uses a direct-mapped cache, both the source and destination will use the same cache line, which will cause a read and write miss for each entry in the line. This is horribly inefficient. Here's an example:

#define CACHESIZE 0x40000

char source[CACHESIZE], target[CACHESIZE];

for (i = 0; i < CACHESIZE; i++) {

   source[i] = target[i];

}

Support exists in processors to accelerate block copying. Sun's VIS instruction set implemented in the UltraSPARC-series processors has a block-copy instruction that bypasses the caches, while still maintaining consistency between the caches and main memory. The bcopy(3C) and memcpy(3C) function calls both use this mechanism on VIS-enabled processors automatically.

3.2.5 The Cache Size Anomaly

Based on what we have said so far, the general rule of "increased cache means increased performance" holds. If a piece of code operates in a cache-busting fashion, it may be that the code performs much faster without a processor cache at all! Even so, sometimes very strange performance results are observed, like a SPARCstation 5 Model 110 outperforming a SPARCstation 20 Model 85 in a large simulation. Why is this the case?

The SPARCstation 5 uses a microSPARC processor that was designed for cheap, efficient operation. It has a small cache, a high cache miss rate, but a very low cache miss cost. The SuperSPARC-II processor used in the SPARCstation 20 has a much larger cache, a much lower cache miss rate, and a much higher cache miss cost. In some applications (such as database processing), the large cache works efficiently, and the SPARCstation 20 is much faster, but in a simulation where the cache is not large enough to help performance at all, the low cache miss cost of the SPARCstation 5 makes it an extremely fast performer. The UltraSPARC design incorporated many of these features, which has a very low latency to main memory.

While I've given an example that uses older hardware, the potential for this sort of problem comes up quite frequently. For example, Celeron processors sometimes outperform their "professional" Pentium-II counterparts, because of a reduced cache miss cost. Some in-depth testing may be necessary to determine exactly what the cache access patterns of your applications are.

Another influence on cache effectiveness is something called page coloring, which refers to the algorithm used by the operating system kernel to decide where pages should be placed in the processor caches. For more information on this topic, see Section 4.2.6 in Chapter 4.


3.3 Process Scheduling

Another factor that affects processor performance is process scheduling. Process scheduling is how the operating system determines which process to run on the CPU. The next four sections look at different process scheduling models used in the Unix world.

3.3.1 The System V Model: The Linux Model

Linux implements a relatively simple, standard Unix scheduling model. It supports process preemption (interrupting a running process with a higher priority process), although its kernel is nonpreemptable.

The Linux kernel scheduling algorithm works by breaking the CPU time into units called epochs. In a given epoch, each process has a specified time quantum whose duration is computed when the epoch begins; therefore, each process can have a different quantum. This quantum represents the most CPU time that is consumable by that process during that epoch. When a process has exhausted its quantum, it is removed from the processor and replaced by another runnable process; of course, a process can be selected by the scheduler several times, as long as its quantum has not been exhausted (for example, it might have voluntarily relinquished the processor to wait for some disk I/O activity to complete).

When the scheduler is deciding what process to run next, it considers the priority of each process. There are two types of priority: static priority and dynamic priority. Static priority is assigned by users to real-time processes, and is never adjusted. Dynamic priority applies only to conventional processes, and is the sum of the base time quantum (also known as the base priority of the process) and the number of ticks of CPU time left to the process before its quantum expires in the current epoch. The static priority of a real-time process is always higher than any dynamic priority: the scheduler will only run a conventional process if there are no runnable real-time processes.

The epoch ends when all runnable processes have exhausted their quantum; the scheduler then recomputes the time quantum durations for all processes and a new epoch begins. The default quantum for each process, if it exhausted its quantum in the prior epoch, is the process's base time quantum. A new process inherits the base time quantum of its parent. Since process 0 (the swapper) has a default base time quantum of 20 ticks (about 200 milliseconds), each process by default has a base time quantum of equal size. The base time quantum value for a specific process is adjustable by the nice(3C) and setpriority(3C) system calls, as well as the nice command.

3.3.1.1 Finding a process's priority

Determining the priority of a running process can be done by observing the PRI column of the output of ps -el, as illustrated in Example 3-1.

Example 3-1. Finding a process's priority in Linux
% ps -el

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

100 S     0     1     0  0  60   0    -   326 do_sel ?        00:00:48 init

040 S     0     2     1  0  69   0    -     0 contex ?        00:00:00 keventd

040 S     0     3     1  0  69   0    -     0 kswapd ?        00:05:32 kswapd

040 S     0     4     1  0  69   0    -     0 krecla ?        00:00:00 kreclaimd

040 S     0     5     1  0  69   0    -     0 bdflus ?        00:24:43 bdflush

040 S     0     6     1  0  69   0    -     0 kupdat ?        00:23:01 kupdate

...

140 S     0 11602   488  0  68   0    -   745 tcp_da ?        00:00:00 sendmail

140 S     0 11727   488  0  68   0    -   745 tcp_da ?        00:00:00 sendmail

140 S     0 11793   488  0  69   0    -   745 tcp_da ?        00:00:00 sendmail

000 R   563 11877  5019  0  75   0    -   759 -      pts/3    00:00:00 ps
3.3.1.2 Adjusting a process's effective priority

Adjusting a process's effective priority is accomplished in a straightforward manner via the renice command. The renice command changes the absolute "niceness" value of the process. This value can range from +20 (the affected processes will only run when the system is otherwise idle) to -20 (the affected processes will be scheduled with priority). The default value is zero.

For example, let's look at a certain process (rhof ):

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S   563  5620  5601  0  69   0    -   514 do_sel pts/6    00:00:00 rhof

Let's decrease the priority of this process to -20:

% renice -20 5620

5620: old priority 0, new priority 20

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S   563  5620  5601  0  69  19    -   514 do_sel pts/6    00:00:00 rhof

Notice that the NI column value has become 19, reflecting the process's decreased priority. Nonroot users cannot increase the priority for their own processes, even if they were the ones who decreased them in the first place:

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S   563  5865  5601  0  70   0    -   514 do_sel pts/6    00:00:00 rhof

% renice +5 5865

5865: old priority 0, new priority 5

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S   563  5865  5601  0  71   5    -   514 do_sel pts/6    00:00:00 rhof

% renice +3 5865

renice: 5865: setpriority: Permission denied

In order to decrease the process's priority, we need to be root:

% sudo renice -20 5865

5865: old priority 0, new priority -20

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S  1000  3274  3240  0  59 -20     -  1619 select pts/1    00:00:00 rhof

This provides a simple, easy way to adjust the priority of a running process.

3.3.1.3 Modifications for SMP systems

In a multiprocessor system, it is possible for an interesting conundrum to arise. Let's say there are two processors: one of them is idle, and the other is executing a process at relatively low priority. On the busy processor, a high priority process becomes executable. Should we execute the high priority process immediately on the idle CPU, in which case we have to do some work to migrate the cache lines for the process between processors, or do we defer it to run on the existing CPU? Linux addresses this issue by adopting an empirical rule related to the processor's cache size: the larger the processor's cache, the longer a process will wait for a piece of time on that processor.

3.3.2 Multilayered Scheduling Classes: The Solaris Model

One of the very nice features of the Solaris operating environment is the ability for administrators to control how the operating system schedules processes for execution. This allows fine-grained control over what processes are running at what priorities. One fundamental thing to keep in mind is that Solaris does not directly schedule processes; rather, it schedules the underlying kernel threads.

3.3.2.1 The Solaris threading model

Before we can continue our discussion of process scheduling, we need to discuss exactly what a kernel thread is. Solaris implements a multilayer threading model, which is intended to decouple the management of user-level threads from the kernel. User-level threads have their own priority scheme, and are scheduled by a scheduling thread that is created by the threading library when a multithreaded application is compiled and executed. This allows multithreaded applications to spawn thousands of threads without significantly loading the kernel; in fact, user threads aren't made visible to the kernel until they are bound to a lightweight process, or LWP, which has some state in the operating system. The scheduling thread is responsible for mapping the user threads into the LWP, which is associated with a kernel thread for execution on a processor.[10] Every LWP has a kernel thread, but every kernel thread need not have an LWP: some kernel threads are used exclusively by the operating system, and thus an LWP is not required.

[10] A nonthreaded application has one LWP and one kernel thread associated with it.

Periodically, some literature indicates that kernel threads and LWPs are the same thing. The truth is that they are distinctly different, but in the discussion of user processes, it's acceptable to think of them as one large entity. There is always, without exception, a one-to-one relationship between LWPs and kernel threads.

For the rest of our discussion of scheduling, the word "thread" will refer to kernel threads specifically.

3.3.2.2 Scheduling classes

Threads generally fall into five categories for scheduling: timesharing (ts), interactive (ia), kernel, real-time (rt), and interrupt. The relative priorities in each of these classes can be mapped into absolute priorities, as is summarized in Table 3-3.

Table 3-3. Process priority table layout in Solaris

Absolute priority range

Relative priority range

Scheduling queue

169-160 (or 109-100)

9-0

Interrupt threads (if real-time scheduling class is not loaded)

159-100

59-0

Real-time priorities

99-60

39-0

Kernel priorities

59-0

59-0

Timesharing and interactive priorities

The timeshare class is the default class that processes are scheduled in. However, the interactive class exists to improve interactive performance on windowing systems. This is accomplished by giving the foreground processes in the active window a higher priority. The processes that are chosen to be in the interactive class are those that come up under the windowing environment; remote processes are still scheduled in the timesharing class. However, the interactive and timesharing scheduling classes overlap in priority, so an interactive process at priority 59 is running at the same priority as a timesharing process at priority 59.

Let's discuss how the operating system decides the threads that should be run.

Notes on Real-Time Scheduling

Real-time scheduling refers to the means by which a process can be assured of having a slice of the processor when it needs it. This is a requirement for certain types of applications, such as robotics and flight control. In general, operating system support for real-time scheduling falls into three categories: none whatsoever, hard real-time support (in which all hard deadlines for applications are met), or soft real-time support (in which deadlines may be missed, but this is kept to a statistical minimum). Solaris is a soft real-time environment.

In a soft real-time environment, the response time must be finitely bounded (e.g., there must be an upper bound on the time during which the application can be off the processor -- in Solaris, this may be as long as five milliseconds). Unix environments typically do not implement real-time support because of two problems. First, page faults can induce unpredictable latency (see Section 4.3). Solaris solves this problem by locking all the pages of a real-time process in memory. The other problem is that in most Unix systems, the kernel is not preemptable, meaning that it can be removed from the processor and another process run. There are certain points at which the kernel cannot be preempted -- these are called nonpreemption points. Only special cases are nonpreemptable: the most common nonpreemptable case is when the kernel is executing in interrupt context. However, a higher priority interrupt can displace an interrupt that is in the process of being serviced.

3.3.2.3 The dispatcher

When a thread is ready to run, it is placed on a dispatch queue. Every processor has a separate set of dispatch queues. This state (waiting on the dispatch queue) is known as the RUN state. When a processor becomes free, the dispatcher takes the thread with the highest priority from the dispatch queue for that processor, and places it on the processor. The thread is then in the ONPROC state, and can perform computation.

A thread can leave the processor by several mechanisms. The first is that it performs computation until it reaches a time limit (determined by its priority); this is called the quantum. The quantum is usually inversely related to the priority (higher priority processes have smaller quanta), since higher priority processes are scheduled more frequently. When this happens, the thread's priority is set to a new level, which is referred to as the tqexp value. This level is almost always lower than the thread's priority when it started; this is done to ensure "fair share" scheduling.

Another way for the thread to leave the processor is if it blocks; for example, by waiting for an I/O to complete or for a lock to be released. This causes the thread to go into the SLEEP state. Threads that go to sleep while holding "critical" resources (such as reader/writer locks) are assigned a priority in the kernel class, so that they will have a chance to release their locks quickly. When processes on the sleep queues are returned to the processor, they have their priority set to a new value, called slpret. This is usually higher than the thread's earlier priority, so that the thread gets some execution time fairly quickly after it's gone to sleep.

The third and final way for a thread to leave the processor is if it is forcibly required to, usually by the kernel preempting the thread. If a preempted thread accrues enough time (determined by the maxwait parameter) of being starved for the processor, its priority is increased to the value specified by the lwait parameter. This provides a way of compensating threads that have been waiting on the dispatch queues for a long time after being preempted.

3.3.2.4 Checking a process's priority

You can find out what lightweight process and scheduling class a specific process is associated with by using the -L and -c switches, respectively, with /usr/bin/ps, as shown in Example 3-2.

Example 3-2. Viewing a process's priority in Solaris
% /usr/bin/ps -efcL

     UID   PID  PPID   LWP  NLWP  CLS PRI    STIME TTY     LTIME CMD

    root     0     0     1     1  SYS  96   Mar 20 ?        0:00 sched

    root     1     0     1     1   TS  58   Mar 20 ?        0:06 /etc/init -

    root     2     0     1     1  SYS  98   Mar 20 ?        0:00 pageout

    root     3     0     1     1  SYS  60   Mar 20 ?       22:37 fsflush

...

    root   200     1     1     7   TS  51   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     2     7   TS  58   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     3     7   TS  59   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     4     7   TS  58   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     5     7   TS  59   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     6     7   TS  59   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     7     7   TS  58   Mar 20 ?        0:01 /usr/sbin/nscd

    root   185     1     1     1   TS  48   Mar 20 ?        0:01 /usr/sbin/cron

...

    root  4897   164     1     1   TS  38 20:28:02 ?        0:00 in.telnetd

    root  5235  4899     1     1   TS  48 21:10:19 pts/1    0:00 /usr/bin/ps -cefL

     gdm  4899  4897     1     1   TS  58 20:28:02 pts/1    0:00 -csh

I have significantly trimmed the output of this command. Of particular interest here are the LWP, CLS, and PRI columns, which represent the lightweight process number, scheduling class, and priority, respectively. Of definite interest is the fact that multithreaded processes (for example, the name service cache daemon, /usr/sbin/nscd ) can exist on multiple LWPs concurrently (the number of LWPs a process is associated with is given by the NWLP column).

3.3.2.5 Tuning the dispatch tables

The dispatch tables are tunable via the dispadmin command. This is accomplished by writing the dispatch table for a specific class to a file, editing the file, and then reloading the file into the dispatch table.

To see what's available for editing, you can display the list of active scheduling classes by dispadmin -l:

% dispadmin -l

CONFIGURED CLASSES

==================

SYS     (System Class)

TS      (Time Sharing)

RT      (Real Time)

IA      (Interactive)

You can then dump the scheduling table by using dispadmin -c class -g:

% dispadmin -c TS -g

# Time Sharing Dispatcher Configuration

RES=1000

# ts_quantum  ts_tqexp  ts_slpret  ts_maxwait ts_lwait  PRIORITY LEVEL

       200         0        50           0        50        #     0

       200         0        50           0        50        #     1

       200         0        50           0        50        #     2

       200         0        50           0        50        #     3

...

       200         0        50           0        50        #     9

       160         0        51           0        51        #    10

       160         1        51           0        51        #    11

       160         2        51           0        51        #    12

       160         3        51           0        51        #    13

       160         4        51           0        51        #    14

       160         5        51           0        51        #    15

...

        40        42        58           0        59        #    52

        40        43        58           0        59        #    53

        40        44        58           0        59        #    54

        40        45        58           0        59        #    55

        40        46        58           0        59        #    56

        40        47        58           0        59        #    57

        40        48        58           0        59        #    58

        20        49        59       32000        59        #    59

The output of this table is significantly trimmed. Let's walk through it piece by piece.

The first line specifies the resolution of the time figures in the table below. The reciprocal of the resolution value is used to interpret the ts_quantum column in the dispatch table: as the value of RES increases, the ts_quantum values increase correspondingly. By default, RES is 1000. The reciprocal of RES is 0.001, which in fractions of a second represents one millisecond. By default, then, the units for the ts_quantum field are milliseconds. While the resolution value is changeable, it is usually advisable to leave it at the default.

I've already described the meanings of most of these columns in the discussions of how the dispatcher works (see Section 3.3.2.3 earlier in this chapter). I'll briefly summarize them here:

  • The ts_quantum field represents the maximum amount of time the thread can remain on the processor.

  • The ts_tqexp field specifies the priority the thread will be assigned following its removal from the processor after it uses its entire time quantum.

  • The ts_slpret field gives the priority a process will have after it returns to execution after going to sleep (e.g., waiting for an I/O to complete).

  • The ts_maxwait and ts_lswait fields control the compensation of a process (set to the priority given by ts_lswait) that has been preempted by a kernel-priority thread for a long period of time (greater than ts_maxwait).

Some features are of particular interest: the first is that the quantum values decrease correspondingly with the priority level. Another is that the maxwait value for priority level 59 is set very high; this ensures that every thread has a fair chance at being scheduled.

Because each priority class has a zero value in ts_maxwait for every priority except the highest level, all but the highest priority timesharing threads will have their priorities readjusted into the 50-59 range once a second or so. This has the effect of not penalizing a thread that is CPU-bound for an extended period. Threads that are CPU hogs will, over time, end up spending most of their time in the 0-9 priority range, as they will keep using up their time quantum and suffer from ts_tqexp priority readjustment. Once every second, they'll be bumped up again, but will only migrate back down if they continue to sustain their CPU-intensive behavior.

The real-time (RT) class has a much simpler table:

# Real Time Dispatcher Configuration

RES=1000

# TIME QUANTUM                    PRIORITY

# (rt_quantum)                      LEVEL

      1000                    #        0

      1000                    #        1

      1000                    #        2

      1000                    #        3

      1000                    #        4

...

       100                    #       55

       100                    #       56

       100                    #       57

       100                    #       58

       100                    #       59

This is simpler because the real-time thread scheduling does not involve any of the complex adjustments described previously. Real-time threads run until they stop or until their quantum expires (in case multiple real-time threads are pending). Also, note that even though the priority level range is from zero to 59, the real-time threads are scheduled at a higher priority than the kernel, as these relative priorities are mapped to absolute priorities via the map discussed previously.

You can tweak the dispatch tables by writing the output of dispadmin to a file (as shown), editing the file, and then loading it by running dispadmin -c class -s file.

The most common technique employed here is to fence off the top timesharing class, so that high priority CPU-intensive jobs run without being forced down to lower priority levels. You can accomplish this by editing the timesharing class dispatch table to look something like what's shown in Example 3-3.

Example 3-3. Creating a priority fence in the timesharing class
# Time Sharing Dispatcher Configuration

RES=1000

# ts_quantum  ts_tqexp  ts_slpret  ts_maxwait ts_lwait  PRIORITY LEVEL

...

        40        36        57           0        57        #    46

        40        37        58           0        58        #    47

        40        38        58           0        58        #    48

        40        39        58           0        58        #    49

        40        40        58           0        58        #    50

        40        41        58           0        58        #    51

        40        42        58           0        58        #    52

        40        43        58           0        58        #    53

        40        44        58           0        58        #    54

        40        45        58           0        58        #    55

        40        46        58           0        58        #    56

        40        47        58           0        58        #    57

        20        48        58       32000        58        #    58

       200        49        59           0        59        #    59

I have highlighted changed values. The net effect is that no process can naturally find its way into priority level 59 (this is why the ts_lwait parameter has been adjusted). Priority level 58 now functions the same as priority level 59: it has an unusually low quantum and a high value of ts_maxwait. By increasing the quantum of priority 59, threads in that class will run longer before they are forced to relinquish the CPU and look for other threads that need to run. In order to get a process into this priority level, you will need to run priocntl, as described in the next section.

There's one small catch with this scheme: you will need to flush any existing processes out of priority level 59. This is accomplished by creating a copy of Example 3-3, and changing the last line (for priority level 59) to this:

# Time Sharing Dispatcher Configuration

RES=1000

# ts_quantum  ts_tqexp  ts_slpret  ts_maxwait ts_lwait  PRIORITY LEVEL

...

       200         0         0           0         0        #    59

This ensures that any process in priority level 59 quickly departs from it. You can then use ps -ecL | grep 59 to find processes that are still running in priority 59; they may need to be stopped and restarted.

Please exercise care when adjusting the dispatch tables. It is possible to cause a great amount of damage to the stability and usability of a production system through this interface. Test thoroughly!

3.3.2.6 Adjusting priorities

You can directly adjust the priority of a process via the priocntl command. This is accomplished by specifying the -s flag, which indicates that you are setting a priority: either -p priority, which specifies the priority level to be set, or -i pid process-id, which specifies the process ID to be adjusted. For example, let's adjust the nscd process into priority level 59, after having adjusted the dispatch tables as described in Example 3-3 to fence off the timesharing priority level 59:

# /usr/bin/ps -ecL | grep 59

# /usr/bin/ps -ecL |/usr/xpg4/bin/grep -E 'nscd|PID'

   PID   LWP  CLS PRI TTY     LTIME CMD

   200     1   TS  51 ?        0:00 nscd

   200     2   TS  58 ?        0:00 nscd

   200     3   TS  58 ?        0:00 nscd

   200     4   TS  58 ?        0:00 nscd

   200     5   TS  58 ?        0:00 nscd

   200     6   TS  58 ?        0:00 nscd

   200     7   TS  58 ?        0:01 nscd

# priocntl -s -p 59 -i pid 200

# /usr/bin/ps -ecL |/usr/xpg4/bin/grep -E 'nscd|PID'

   PID   LWP  CLS PRI TTY     LTIME CMD

   200     1   TS  59 ?        0:00 nscd

   200     2   TS  59 ?        0:00 nscd

   200     3   TS  59 ?        0:00 nscd

   200     4   TS  59 ?        0:00 nscd

   200     5   TS  59 ?        0:00 nscd

   200     6   TS  59 ?        0:00 nscd

   200     7   TS  59 ?        0:01 nscd

The advantage of a trick like this is to ensure that high priority jobs get as much processor time as possible, regardless of the environment in which they are running.

The renice command, which we discussed earlier in this chapter in Section 3.3.1.2, provides another way of adjusting a process's priority, by providing a static offset into the priority tables. This offset is equal to the renice amount, but with the negative sign reversed. For example, a process that has been reniced to -20 recieves a +20 offset into the priority table (so its priority will never drop below 20).


3.4 Multiprocessing

Shared-memory multiprocessor systems were considered rather exotic in the mid-1980s, and they were expensive. Since then, we have seen hardware costs drop, much more robust multiprocessor support in operating systems, and an increased demand for affordable high-throughput systems at a workgroup and even desktop level. These multiprocessor systems are described as uniform memory access (UMA) architecture, which means that all physical memory is accessible to all processors at the same rate. Some large-scale systems use a nonuniform memory access (NUMA) architecture, in which a specific processor can access some bits of physical memory faster than others. Figure 3-5 illustrates the UMA and NUMA architectures.

Figure 3-5. UMA and NUMA architectures

We'll restrict our discussion here to UMA architectures.

3.4.1 Processor Communication

In order for the human body to function nominally, it is vital that information can be conveyed to and from the brain. Communication is also essential for successful operation of a multiprocessor system. If we draw an analogy between computers and the human body, and we think of the processor as the brain, the "spinal cord" facilitates the exchange of information to and from the processor to main memory and peripheral devices. This is facilitated in computer architecture by buses or crossbars, which also allow the arbitration of communication so that the components can figure out who can talk at any given point in time. It is important to note that we speak here only of interconnections between processors and main memory; for details on how interconnections between peripherals are handled, see Section 3.5 later in this chapter.

3.4.1.1 Buses

A bus consists of three parts: a protocol for communication, a set of parallel wires that link all the system components, and some supporting hardware. Buses are cheap and easy to build, but as it becomes more heavily populated and the load increases, the bus begins to be a performance bottleneck. As a result, cache size and performance is very important in bus-based multiprocessor configurations. Typically, this sort of architecture supports at most four processors, but larger ones will support a few tens of processors. It is possible that the bus or memory subsystem will become saturated before the maximum number of supported processors have been installed!

Buses implement either circuit-switched or packet-switched protocols. In a circuit-switched protocol, two devices establish a communications circuit over the bus and maintain that circuit until their transaction has completed. This is simple to implement, but the processor must wait if it is communicating with a slow device (such as main memory). These wait states reduce the bus throughput and waste clock cycles: the faster the processor clock rate, the more cycles wasted.

Older Sun systems used the MBus bus architecture, which is circuit-based.[11] You could actually configure four 50 MHz SuperSPARC modules with SuperCache in a SPARCserver 20, which was a remarkably fast desktop system for the time. Unfortunately, when 50 MHz processors were installed in a SPARCstation 20, it automatically reduced the MBus clock rate from 50 MHz to 40 MHz in order to support data synchronization between processors; it's too bad that the system that needed the fastest possible MBus didn't get it. Because of cooling limits in the SPARCstation 20 chassis, you could not configure processors faster than 50 MHz.

[11] The SPARCstation 10, 20, and the SPARCserver 6x0MP. Other systems, such as the SPARCstation LX, Classic, 4, and 5, are MBus-based, but do not have multiprocessor support. If uname -a returns sun4m, your machine uses the MBus architecture.

To get around the wasted cycles inherent in fast-slow device interactions on a circuit switched protocol like MBus, Sun implemented a packet-switched protocol for XBus, which was used on a per-board basis in the SPARCserver 1000 and 2000 lines as well as in the Cray SuperServer CS6400.[12] The boards were tied together via XDBus, which consisted of multiple interleaved XBuses. A packet-switched protocol relies upon breaking every bus transaction into a request packet and a response packet. This is hard to implement because every packet must contain an identifier of the issuing device, as well as the need for a particular component on the bus (such as the memory subsystem) to queue up requests if they come in too quickly.

[12] When Cray was purchased by Silicon Graphics in 1996, the division responsible for the SuperServer CS6400 was sold to Sun. This group later produced the Ultra Enterprise 10000 ("Starfire") system.

With the advent of the fast UltraSPARC processors, Sun designed a new interconnect architecture, called the UltraSPARC Port Architecture switched interconnect (UPA). It is a packet-switched architecture characterized by its implementation of a "switch" architecture in which all buses route through a centralized switch that redirects them appropriately, as well as sizing the bus to the device; for example, memory devices require a very wide bus, but I/O channels need not be so wide. This bus sizing keeps costs low. All ports on the switch run at the same clock rate, but some are master ports (processors, I/O controllers) that can both initiate and respond to requests, and some are slave ports (framebuffers, memory) that can only respond to requests. UPA is resistant to bus contention bottlenecks under heavy load, because each device has its own dedicated connection to the bus switch.

For the multiboard Ultra Enterprise server line, modifications were made to the UPA bus in order to support the multiboard system requirements.[13] This gave rise to the Gigaplane passive backplane, which is a distributed extension of the UPA bus techniques. It has support for optimizations that increase the efficiency of spinlocks and memory sharing.

[13] The Ultra Enterprise 250 and Ultra Enterprise 450 systems are single-board, and implement UPA.

The performance of these buses is detailed in Table 3-4.

Table 3-4. Sun multiprocessor bus/switch characteristics

Bus

Clock rate

Peak bandwidth (in MB/sec)

Peak read throughput (in MB/sec)

Peak write throughput (in MB/sec)

MBus

40 MHz

320

90

200

XBus (per board)

40 MHz

320

250

250

Single XDBus (SPARCserver 1000)

40 MHz

320

250

250

Dual XDBus (SPARCcenter 2000)

40 MHz

640

500

500

Quad XDBus (SuperServer CS6400)

55 MHz

1,760

1,375

1,375

UPA (UltraSPARC systems)

82 MHz

1,600

1,300

1,300

Gigaplane (Ultra Enterprise systems)

82 MHz

2,624

1,600

1,600

Gigaplane XB (Ultra Enterprise 10000)

100 MHz

Up to 10,622

Up to 12,800

Up to 12,800

3.4.1.2 Crossbars

Crossbars were developed to work around the performance problems in the single bus architecture. A crossbar is essentially a collection of several parallel buses, in which any component can access any other component through any path through the crossbar. In addition to connecting parties that wish to communicate, much like a switchboard operator, the crossbar must also arbitrate conflicts, such as those that arise when two processors try to access the same bit of memory. Because there is no single shared bus, performance is generally good, but because of the extra support hardware required, crossbars are typically more expensive, and only found in high-end multiprocessor systems.

One example of such an implementation is the Gigaplane XB crossbar used in the Sun Ultra Enterprise 10000 system. It is an extension of the Gigaplane passive backplane used in the other Ultra Enterprise systems; it increases the throughput of the architecture, but at a cost in price and memory latency. This increase in memory latency is caused by an additional layer of circuitry. Gigaplane XB does not feature the spinlock optimizations included in Gigaplane, because Gigaplane XB was developed by the Cray SuperServer team before their acquisition. Gigaplane is actually faster than Gigaplane XB when running within its bandwidth limitations; the full bandwidth of the XB crossbar is only available with a fully configured system, as illustrated in Table 3-5.

Table 3-5. Bandwidth variation as a function of configured processors

Architecture

Number of processors

Peak bandwidth (in MB/sec)

Latency (in ns)

UPA

1-4

1,600

170

Gigaplane

1-30

2,624

240

Gigaplane XB

1-12

13-20

21-28

29-36

37-44

45-52

53-60

61-64

1,333

2,666

4,000

5,331

6,664

7,997

9,330

10,622

400

In less than twenty processor configurations, the Gigaplane architecture has the edge in performance. In larger installations, the Gigaplane XB crossbar starts to shine, and we see much improved bandwidth: four times the bandwidth for twice the number of configured processors. In small installations (four processors or less), however, the latency of Gigaplane-based systems becomes a liability; because of the increased complexity of the switching times, it is approximately twice that of UPA-based systems. This means a four-processor single-board Ultra Enterprise server often outperforms a four-processor multiboard Ultra Enterprise server, and costs considerably less!

3.4.1.3 UltraSPARC-III systems: Fireplane

With the move to the UltraSPARC-III architecture, Sun developed a new interconnection technology, called Fireplane. Fireplane improves on the previous-generation UPA interconnect in both performance and feature set. Unlike the other interconnects we've discussed, Fireplane does not specify the device topology; instead, it describes the protocols that devices attached to the interconnect use to communicate. This gives it a good degree of flexibility for future uses. The other interesting new feature of Fireplane is that it supports having multiple outstanding transactions on the interconnect at once (up to fifteen 64-byte data transfer requests, for example, can be pending at once on the Sun Blade 1000).

The Sun Blade 1000 implements a packet-switched bus topology for both address and data paths. The data path is a 288-bit (256 bits of data, plus 32 bits of error correction information) bus, implemented via a group of six chips that are collectively known as the combined processor memory switch (CPMS). There are point-to-point connections between data path and devices connected to it, simplifying arbitration logic.

The memory controller is integrated onto the UltraSPARC-III processor, which reduces latency and improves bandwidth. The memory is addresed through a 144-bit-wide data path to a data switch chip, which has a 576-bit (512 bits of data, plus 64 bits of error correction information) path to memory. Interestingly, in a two-processor system, only one of the processors' memory controllers is active. This processor can directly access memory, whereas the other must access memory over the Fireplane interconnect.[14]

[14] All memory address requests go across the interconnect to facilitate cache coherency management.

The interconnect itself is capable of a peak bandwidth of 67.2 GB/second, with a sustained bandwidth of up to 9.6 GB/second (2.4 GB/second per CPU, up to a limit of 9.6 GB/second).

3.4.1.4 "Interconnectionless" architectures

Some UltraSPARC processors, which are typically used in lower-end systems, do not require a formal interconnect. These include the so-called "I-series" and "E-series" designs, such as the UltraSPARC-IIi and the UltraSPARC-IIe; I-series processors are highly integrated and optimized for price/performance, whereas the E-series processors are designed for embedded applications.

3.4.2 Operating System Multiprocessing

The majority of modern operating systems support multitasking. In fact, multitasking does not require more than one processor; it simply refers to the ability to run more than one process at a time. The system splits its time between them on the basis of context switching at fixed time intervals, I/O activity, or interrupts. On a multiprocessor system, individual processes will be split between both processors for an aggregate speedup (but without increasing the observed single-application performance). Most processes are self-contained,[15] but processes can play nicely with others and share information by means of the interprocess communication (IPC) mechanisms such as pipes and sockets or by intimately shared memory (ISM). We'll discuss ISM in greater detail in Section 4.4. It is generally a bad idea to implement IPC in an attempt to increase single-application performance on a UMA multiprocessor system; on very large scale parallel computing installations it may be appropriate, however.

[15] In terms of not sharing data with other processes. We won't worry about shared libraries and such for the purposes of this discussion.

3.4.3 Threads

A thread is not a process! A process begins its life as a single thread, but it can then add or remove threads as its execution progresses. These threads all share the same process memory space, but each thread also possesses a thread private area for its own local variables. Threads can improve single-application performance in two ways:

  • Each thread in a multithreaded process can be dispatched to a different processor in a multiprocessor system. This collaboration across multiple processors improves single-application performance.

  • Threads can be used to increase application throughput. A single-threaded web server would be stuck in a wait state every time it needed to fetch a document from disk. By multithreading the server, one thread can be waiting to handle a new request while another waits for the disk to provide data.

Because of this last point, it is quite practical to have a multithreaded process on a single processor; in fact, multithreading existed long before multiprocessing systems did. This technique is very useful for increasing throughput, but it's a bad idea to have more identical active threads than processors in a compute-intensive program.

3.4.4 Locking

It is critical to the smooth operation of a multiprocessor system that some mechanism of synchronization should exist. For example, say that we have two threads that each increment the same variable by executing the following assembler code:

LOAD      r1,Counter

ADD       r1,1

STORE     r1,Counter

Let's say that the initial value of Counter is 021400. What happens if, at the exact instant that the first thread completed the LOAD but before it executed the ADD and STORE, the operating system switched to executing the second thread? The second thread could execute all three instructions, resulting in Counter being 021401. When the operating system restarts the first thread, it will pick up where it left off and execute the ADD and STORE instructions, storing 021401 into Counter and giving it a terminal value of 021401 when it should be 021402. If the system has two processors, the situation is just as bad if not worse! Rather than relying on operating system intervention, if the two threads are executing the same instructions concurrently, they will perform the load-add-store sequence in perfect lockstep, and we will again have 021401 when we should have 021402 in Counter.

We need some way to make simple operations such as this atomic (indivisible), so that only one thread can execute these instructions at a time. We call areas like this critical sections.

On uniprocessor systems, this problem was solved by disabling interrupt servicing for the duration of the critical section, which prevents the thread from being interrupted. This won't work on multiprocessor systems, since each processor has its own interrupt mask, so we must be more devious in our approach.

All SPARC processors support an instruction called LDSTUB (load-store-unsigned-byte), which atomically reads a byte from memory into a register and writes 0xFF into memory. This instruction allows the establishment of mutual exclusion locks, or mutexes, which make sure that only one processor at a time can hold the lock. To tear down the lock, the thread simply writes 0x00 back to memory. If a processor can't get a lock, it may decide to spin until the lock becomes available. Spin locking is accomplished entirely within the processor's cache, so that it does not cause excess bus traffic. Spin locks are useful when the wait for the lock is expected to be short; if a longer wait is expected, the thread should sleep so that another job can be scheduled onto the processor. The SPARC V9 instruction set used on UltraSPARC processors includes an atomic compare-and-swap operation as well as the atomic load-store operation.

When the multiprocessing-enabled version of SunOS 4 was released for the SPARCserver 6x0-class systems, the uniprocessor interrupt-mask approach was replaced with a mutex-based approach: the call to disable interrupts was replaced by a mutex acquisition, and the call to re-enable interrupts was replaced with a mutex release. On a system with many processors that is under load, the mutex locking itself can actually become a bottleneck.[16] If this is the case, adding additional processors will hinder, rather than help, performance.

[16] I've never seen this behavior on a system with less than four CPUs, assuming a well-behaved application.

Interestingly, this phenomenon of performance decreasing as you add processors is not restricted to this particular case. It's possible for poor application programming to cause severe mutex contention. A large security company a friend of mine was affiliated with had a product written in C that ran on Solaris systems. On a uniprocessor system, this product performed adequately. When a second CPU was added, the performance of the application dropped by almost 30%. This is not at all unusual: poor application design and implementation is a possible, indeed a likely, root cause of poor performance on multiprocessor systems.

Sometimes all the threads in a process need to wait at a certain point until all of the threads are "caught up." For example, let's say that we have a time-based simulation of a car's frontend crumpling during an impact with a concrete wall.[17] It would not make sense for the threads to compute the state of the car at different points in time, since the threads working later would need to know what happened earlier. We can enforce this sort of synchronization with a construct known as a barrier. All threads must reach the barrier (say, finish processing one time interval's worth of data) before any are allowed to continue.

[17] I have, unfortunately, had occasion to attempt this experiment in real life. My personal experience indicates that simulations are a far less expensive way of determining that a car will protect its occupants from bodily harm.

3.4.5 Cache Influences on Multiprocessor Performance

Another major problem needs to be dealt with in multiprocessor system architecture. Let's say that we have two processors, CPU 0 and CPU 1. CPU 0 fetches a specific cache line from main memory into its cache. CPU 1 then fetches the same line into its cache. When CPU 0 writes to that cache line, it will update its local cache as well as main memory; however, there needs to be some way of notifying CPU 1 that its copy of the cache line is invalid and needs to be thrown away. This is accomplished via communications on the multiprocessor bus. Since communication must occur for most cache activity, it creates a good deal of traffic.

As a result, larger processor caches often cause a great performance increase on multiprocessor systems. References to main memory must go across the bus, which on a multiprocessor system is typically very busy.


3.5 Peripheral Interconnects

The last means of interconnection that we need to discuss is between the bus or crossbar that facilitates the communication between processors and main memory (see Section 3.4.1 earlier in this chapter) and the bus that interacts with peripheral devices.

There are two bus architectures in widespread use: Sun's SBus and the industry-standard Peripheral Component Interconnect, or PCI. SBus is being phased out in favor of the PCI standard, but I discuss it here in light of its very large installed base.

3.5.1 SBus

SBus and PCI are fairly similar architecturally: both are designed to be I/O buses (rather than having more general applications), have small form factors requiring a high degree of integration, and roughly similar performance characteristics. If PCI had been available in 1988 when Sun was developing SBus for the SPARCstation 1, SBus probably would not have been developed. Every Sun system implements SBus slightly differently, making a different set of choices between performance, cost, ease of implementation, etc. However, in practice, all current SBus implementations can meet almost all requirements.

SBus is a parallel-transfer (many bits are transferred concurrently) bus architecture. In the original specification, there were 32 address lines and 32 data lines; the Rev B.0 SBus specification permits the address lines to be used during the data cycle to transfer 64 bits; this is implemented in the SPARCstation 10SX, the SPARCstation 20, and all the UltraSPARC-based systems. This arrangement means that 64-bit and 32-bit SBus implementations share the same form factor. However, bus width (while an important factor) is often overestimated in its relevance to performance; very few devices can exceed the 32-bit bus capacity. These are largely very high performance graphics framebuffers and very fast network interfaces (faster than about 250 Mbits/second).

3.5.1.1 Clock speed

In most older Sun systems, the SBus clock is derived from the system clock, either directly or through a divider. Most SPARCstation 10 systems drive the system clock at 40 MHz, and the SBus clock is set at 20 MHz.[18] SPARCstation 20 systems operate the system clock at 50 MHz and the SBus at 25 MHz. The SPARCserver 1000 and SPARCcenter 2000 run the SBus at 20 MHz, independent of any other factors. The microSPARC and microSPARC-II systems run the SBus by dividing the processor clock. This is summarized in Table 3-6.

[18] The exception is in the early models (the SPARCstation 10 Model 20 and Model 30), which had system clocks of 33 MHz and 36 MHz respectively; the SBus clocks are set, then, to 16.5 Mhz or 18 MHz, respectively. Most of these systems have been upgraded to the newer revisions of the SPARCstation 10 boards.

Table 3-6. SBus clock rates for microSPARC systems

CPU clock

Divider

SBus clock

50 MHz

2:1

25 MHz

70 MHz

3:1

23.3 MHz

85 MHz

4:1

21.25 MHz

110 MHz

5:1

22 MHz

Most newer Sun systems (SPARCserver 1000E, SPARCcenter 2000E, and the UltraSPARC-based systems) drive the SBus at 25 MHz, regardless of any other factors.

3.5.1.2 Burst transfer size

The SBus architecture allows data to be transferred in bursts, which are defined as the amount of data that the bus can accept each time a unit on the bus arbitrates for the bus; it is roughly similar to the maximum transmission unit size on a network. Every time a data transfer is initiated, the bus interface hardware accepts one burst's worth of data; it then arbitrates for control of the bus, transmits the target address, and transfers as much data per clock cycle as the bus width allows. This influences the efficiency of bus transactions: on a platform that implements a 16-byte burst size, transferring 2 KB on the bus requires 128 bus arbitration/address/data transfer cycles, compared to just 32 of these cycles when using a 64-byte burst size.

SBus bursts are defined to be 1, 2, 4, 8, 16, 32, and 64 bytes in length. UltraSPARC systems and later MBus-based desktop systems also support a 128-byte burst length. The maximum supported burst size, however, depends on the hardware platform. The SPARCstation 2, SPARCstation 10, SPARCstation 20, and the SPARCsystem 600MP all support 32-byte bursts; all other non-UltraSPARC systems support 16-byte bursts. The UltraSPARC systems support all burst sizes. The burst size is negotiated at boot time, on a per-slot basis, to be the smaller of the maximum size supported by the host SBus device and the controller.

3.5.1.3 Transfer mode

Every SBus implementation offers at least programmed I/O (PIO) and consistent mode direct virtual memory access (DVMA).[19] In PIO, the processor uses load and store instructions to transfer data on the bus. This is primarily used for managing control and status registers on the SBus boards, but a handful of boards implement PIO for data transfer. Because it requires the processor's intervention and because the burst sizes are necessarily smaller (non-UltraSPARC processors define load and store instructions in quantities up to 8 bytes only; UltraSPARC processors have 64-byte block load/store instructions), PIO is almost always more expensive than DVMA. DVMA, however, has a fairly complex setup and teardown process for each I/O operation; therefore, small transfers are more efficiently handled by PIO.

[19] DVMA is very similar to the more common direct memory access (DMA) method. They differ only in that DVMA specifies the address of the memory transaction in virtual space, rather than the physical address.

Some SBus implementations provide an additional transfer mode, called streaming mode DVMA. Streaming mode DVMA provides significantly faster transfers, especially for reads; however, it requires explicit modifications to the driver for keeping transfer hardware consistent with shared memory. Most device drivers implement streaming mode support.

3.5.1.4 Summary of SBus implementations

Table 3-7 briefly summarizes the SBus implementations in many SPARC-based systems.

Table 3-7. SBus implementation details

Platform

Burst size

Bus width

Clock

Streaming?

Speed (read/write)

SPARCstation 1

16 bytes

32 bits

20 MHz

No

12/20 MB/sec

SPARCstation 1+, IPC

16 bytes

32 bits

25 MHz

No

15/25 MB/sec

SPARCstation 2, IPX

32 bytes

32 bits

20 MHz

No

15/32 MB/sec

SPARCstation 10, SPARCstation 600

32 bytes

32 bits

20 MHz

No

32/52 MB/sec

LX, Classic

16 bytes

32 bits

25 MHz

No

17/27 MB/sec

SPARCstation 4, SPARCstation 5

16 bytes

32 bits

21.25-25 MHz

No

36/55 MB/sec

SPARCstation 10 SX

128 bytes

32/64 bits

20 MHz

No

40/95 MB/sec

SPARCstation 20

128 bytes

32/64 bits

25 MHz

No

62/100 MB/sec

SPARCserver 1000, SPARCcenter 2000

64 bytes

32 bits

20 MHz

Yes

45/50 MB/sec

SPARCserver 1000E, SPARCcenter 2000E

64 bytes

32 bits

25 MHz

Yes

56/62 MB/sec

All UltraSPARC systems

128 bytes

32/64 bits

25 MHz

Yes

90/120 MB/sec

The SBus I/O boards used for the Ultra Enterprise series of systems are separate from the processor/memory boards. There are two types of SBus I/O boards; one is designed for graphics cards, whereas the other is more general purpose.

The general-purpose I/O SBus has a dual-ported Fibre Channel interface, a Fast Ethernet port, and a single-ended Fast Wide SCSI-2 port, as well as three 64-bit SBus slots. The board implements two independent SBuses: one is responsible for the Fast Ethernet and SCSI interfaces, along with one of the SBus slots. The other SBus services the Fibre Channel interfaces and the other two SBus slots.[20] The graphics I/O board replaces one of the SBus slots with a UPA slot; this means there is only a single SBus controller. Both the SBus slots are attached to this SBus, along with all the devices integrated into the I/O board. The UPA port is totally independent.

[20] There are actually three types of general-purpose SBus I/O cards. One type (501-4287) has a pair of 25 MB/s Fibre Channel interfaces, and supports only backplane speeds up to 83 MHz. Both of the other types have 100 MB/s Fibre Channel interfaces, implemented via GBIC slots; the only difference between these two is that one (501-4266) runs at 83 MHz, and the other (501-4883) can go up to 100 MHz. However, they all have two independent SBuses, which are broken up as described.

3.5.1.5 SBus card utilization

Table 3-8 provides a general idea of SBus cards and how much bandwidth they typically consume. Of particular interest is the fact that SBus framebuffers are very significant consumers of SBus bandwidth.

Table 3-8. SBus bandwidth utilization by card

Description

Typical bandwidth

Transfer type

Width

Fast/Wide SCSI-2

8-10 MB/sec

DVMA

32

25 MB/s Fibre Channel

16 MB/sec

DVMA

64

Fast/Wide SCSI-2 + Fast Ethernet (hme)

12 MB/sec

DVMA

64

155 Mb/s ATM (Revision 1)

10 MB/sec

DVMA

32

155 Mb/s ATM (Revision 2)

10 MB/sec

DVMA

64

655 Mb/s ATM

50-100 MB/sec

DVMA

64

Fast Ethernet (Revision 2)[21]

4 MB/sec

DVMA

64

Quad Fast Ethernet

20 MB/sec

DVMA

64

Gigabit Ethernet

40 MB/sec

DVMA

64

100 Mb/s FDDI

6-8 MB/sec

DVMA

32

16 Mb/s Token Ring

1 MB/sec

PIO

32

GX, GX+ framebuffer

4-12 MB/sec

PIO

32

TurboGX, TurboGX+ framebuffers

8-20 MB/sec

PIO

32

ZX framebuffers

20-30 MB/sec

DVMA

32

[21] Revision 1 Fast Ethernet cards, which use the be driver, are not supported past Solaris 2.6. If possible, replace them with Revision 2 cards, which use the hme driver.

An SBus should typically not be more than 80 percent utilized, in order to avoid bus contention and arbitration problems that may degrade performance.

3.5.2 PCI

PCI, or the Peripheral Component Interconnect bus, is probably the most widespread bus design in modern computer systems. It was initially introduced into the Intel-based PC world in 1994, and has largely replaced all the older peripheral bus architectures. It is now widely supported by workstation vendors as well. PCI is a standard under the auspices of the PCI Special Interest Group, or PCI SIG (http://www.pcisig.com).

PCI is a synchronous bus architecture, which means that all data transfers are performed relative to a system clock. The initial PCI specification permitted a maximum clock rate of 33 MHz, but the later Revision 2.1 specification extended this to 66 MHz. Most personal computers support only the 33 MHz standard; the primary application of 66 MHz slots is in very high speed networking (e.g., full-duplex Gigabit Ethernet, which greatly benefits from a 66 MHz slot speed). PCI supports a 32-bit multiplexed address and data bus,[22] as well as having architectural support for a 64-bit data bus through a longer connector slot; as with 66 MHz support, 64-bit cards are not widely available in the PC market (see Figure 3-6).

[22] This multiplexing allows for a reduced pin count on the PCI connector, which translates to a smaller physical size and lower cost. Typical 32-bit PCI boards use about 50 pins.

Figure 3-6. PCI connector types

The high speed of the PCI bus (up to 528 MB/second, at 64-bit data paths and a 66 MHz clock rate) limits the number of expansion slots on a single bus to no more than three or four slots due to electrical concerns. In order to permit an expansion bus to have more slots, the PCI SIG has defined a PCI-to-PCI bridge mechanism in PCI Revision 2.1. These bridges are chips that electrically isolate two PCI buses, while still allowing bus transfers to be forwarded from one bus to the other. Multiple bridge devices can be cascades to create a system with many PCI buses.

Multiboard Sun Enterprise systems support the PCI I/O board. These support two PCI cards per I/O board. Each PCI slot is on a separate PCI bus, and the Fast Ethernet and Fast/Wide SCSI interfaces are also on separate PCI buses.

3.5.2.1 PCI bus transactions

In PCI terminology, data is transferred between an initiator, which is the bus master, and a target, which is the bus slave. After the initiator arbitrates for the bus, the transfer itself consists of one address phase and any number of data phases. During the address phase of the transfer, the initiator signals what type of transfer is being done (read from memory, write to memory, I/O read, I/O write, etc.). Either the initiator or the target can insert wait states into the transfer; in addition, the initiator or target may terminate the transfer at any time.

PCI also supports a well-defined automatic configuration mechanism. Each PCI device includes a set of registers that contain configuration data. These registers define what the type of the card is (SCSI, Ethernet, a framebuffer, etc.), as well as who manufactured the card, what the interrupt level of the card is, and so on.

Finally, PCI supports both 5-volt and 3.3-volt signaling levels. However, most early PCI implementations were 5-volt only, and did not provide power on the 3.3-volt pins. Over more time, the 3.3-volt interface will be used more heavily. A "keying" scheme is incorporated in the PCI connector standard, in order to prevent inserting a board into a slot that doesn't supply the correct voltage.

3.5.2.2 CompactPCI

One variant of PCI, CompactPCI, was designed as an industrial-use bus for applications such as real-time machine control and data acquisition, instrumentation, and telecommunications. Compared to the standard, desktop PCI interface, CompactPCI supports twice as many slots (eight versus four), and offers a significantly more robust form factor. CompactPCI cards are designed to front-load into a card cage; this, along with more reliable pin-and-socket connectors, takes full advantage of CompactPCI's hot swap capability. CompactPCI devices should perform equivalently to comparable PCI devices. However, as CompactPCI cards are just beginning to be introduced into the wider workstation/server marketplace, their long term future is unclear.

3.5.3 A Summary of Peripheral Interconnects

Table 3-9 provides a summary of peripheral interconnects; the emphasis is on the PC hardware arena.

Table 3-9. Peripheral interconnect comparison

Bus type

Year

Width

Clock rate

Peak bandwidth

Burst transfers?

ISA

1984

16 bits

8 MHz

8 MB/sec

No

MCA

1987

32 bits

10 MHz

40 MB/sec

Yes

EISA

1988

32 bits

8 MHz

32 MB/sec

Yes

SBus

1988

32 or 64 bits

Up to 25 MHz

Up to 200 MB/sec

Yes

VESA

1992

32 bits

33 MHz

132 MB/sec

Yes

PCI

1992

32 or 64 bits

33 or 66 MHz

132 to 534 MB/sec

Yes

3.5.4 Interrupts in Linux

You can get a list of all the interrupts in use on a Linux system by inspecting the file /proc/interrupts:

% cat /proc/interrupts

           CPU0

  0:  216651906          XT-PIC  timer

  1:       1965          XT-PIC  keyboard

  2:          0          XT-PIC  cascade

  8:          1          XT-PIC  rtc

  9:   96370488          XT-PIC  eth0

 14:  381136978          XT-PIC  ide0

 15:   60751177          XT-PIC  ide1

NMI:          0

ERR:          0

The most useful way to adjust the interrupt distribution in a Linux system is the irqtune application. This application, which is available through http://www.best.com/~cae/irqtune/ at the time of this writing, allows you to specify certain interrupts that are serviced at higher priority. Interrupt priority is used to determine which interrupts are handled first when multiple interrupts occur concurrently. In general, interrupt priorities are assigned in decreasing order of IRQ; that is, the system timer (IRQ 0) has priority over all other IRQs. irqtune addresses this problem by giving priority 0 to the specified IRQs. To invoke it, just run irqtune and list the interrupts to prioritize. For example, to prioritize IRQs 9 and 14 (the eth0 and ide0 interfaces in the system described previously):

# irqtune 9 14

3.5.5 Interrupts in Solaris

Sun enterprise-class systems (the SPARCserver 1000, the SPARCcenter 2000, and Ultra Enterprise series) have many options for managing SBus devices. Since Solaris 2.3, the operating system has used a static interrupt distribution mechanism. This means that each I/O device is assigned to a specific processor; every time that device triggers an interrupt, the specific CPU that is assigned to the device handles the work. The interrupt assignment table is set up at boot time and every time a processor is set to, or removed from, the no-interrupt state (see Section 3.6.7 later in this chapter). Currently, there is no way to manually assign the interrupts from a specific device to a specific processor. The alternative, known as round robin distribution, causes each interrupt to be handled by the "next" processor. It is enabled by setting the do_robin kernel tunable to 1.

Round robin distribution causes the processor caches to be utilized much less efficiently, because a copy of the device-specific interrupt handling code must reside in each processor's cache, but it also results in interrupts being scheduled faster.

Static interrupt processing generally means that high-performance boards should be separated across as many processors and SBus interfaces as possible. The Quad Fast Ethernet (QFE) boards are particularly known for throwing many, many interrupts (up to about 20,000 per second under heavy load). Within an SBus, each slot is treated identically. Because of this, placing multiple instances of an SBus card on the same SBus can sometimes improve performance, because identical boards share the same interrupt processing code. On the other hand, you can easily overload the bandwidth capacity of the SBus. This arrangement improves the chances that the interrupt handling code is resident in the processor's cache. Also note that since interrupts are statically assigned, heavily loading a processor can restrict I/O performance from devices assigned to that processor. If one CPU is taking a large number of interrupts and that CPU is regularly at or near 100% utilization, moving SBus boards into another SBus may provide a significant boost. A simpler solution to try is to enable round robin scheduling.

One common approach to managing interrupt distributions in multiple-CPU systems is to use a small number of CPUs to handle all interrupts, and forbid other processors from taking interrupts. This is accomplished by using the psradm command to put processors into no-intr mode (see Section 3.6.7 later in this chapter). This command lets you segment your workload at a fairly fine level.


3.6 Processor Performance Tools

Concretely detecting whether a system is short of processor time is often more complicated than it first seems. Regular indications of CPU shortage can be attributed to many things: disk configuration problems, peripheral devices being bound to a single processor for interrupt management, processor starvation due to a lack of memory, etc. In a sense, the best way to tell when you're legitimately processor-bound is that you can exclude all other possibilities.

There is also a philosophical issue involved. Some people like seeing a low load average on their systems, or more than 50% of the processor idle. This makes some sense in terms of the system accomodating bursting load activity. Similarly, some people like seeing a load average that approaches the number of processors, meaning the system is running full-tilt. This also makes some sense, because you are using everything that you have paid for in terms of processor performance. My personal philosophy varies between the following: on a computational server, I would aim for full utilization of all processors; on a development system or a database server, I would aim for a degree of overcapacity to accomodate spikes in load.

There are quite a few tools for monitoring processor performance: the load average, vmstat, mpstat, prstat and/or top, and lockstat. There are also two tools in Solaris for managing processors in multiprocessor systems: psrinfo and psrset. We'll cover all of these in some detail.

3.6.1 The Load Average

The system understands a concept of the load average, which is expressed in the uptime command:

% uptime

  4:02pm  up 19 day(s),  9:28,  119 users,  load average: 1.22, 1.21, 1.25

The load average is represented as the average number of runnable jobs (the sum of the run queue length and the number of jobs currently running) over the last one, five, and fifteen minutes. This gives a quick estimate of how heavily loaded the system is, but it is not exclusively a measurement of processor load; a system that has a memory shortage will usually have a high load average.

The load average statistic is only meaningful when looked at in the context of the number of installed processors. As a general rule, if the load average is less than the number of installed processors, the system is not lacking processor power. For example, if the load average is 2.5 on a four-processor system, it is not an indication of a processor shortage.

3.6.2 Process Queues

One very powerful tool in our performance tuning arsenal is vmstat. I discuss it in pieces as we go along. Let's take a look at some sample vmstat output from a Solaris system (recall that vmstat takes as its first argument the time interval in seconds between displaying data, and that you must always discard the first line of output):

% vmstat 30

 procs     memory            page            disk          faults      cpu

 r b w   swap  free  re  mf pi po fr de sr s0 s1 -- --   in   sy   cs us sy id

 0 0 0   5928  1920   0 328 45 25 37 1360 2 4  8  0  0  306 1682  299 10  5 85

 0 0 0 1202648 17152  0 904  8  0  0 1200 0 7 36  0  0  547 5632  683 18 15 67

 0 0 0 1202744 19600  0 1136 0  0  0 1040 0 2 26  0  0  575 4542  752 33 12 55

The first column (highlighted) provides the state of the system's three process queues. In order, they are the run queue (r), the runnable-but-blocked queue (b) and the runnable-but-swapped queue (w). We'll discuss the swap queue's relevance, or lack thereof, in Section 4.5.2.1.

The run queue counts the number of processes that are runnable, but are not currently running. In this example, the system's processor is not overworked. As a general rule, if the run queue is consistently at or more than four times the number of configured processors in the system, you should probably consider increasing the available processors.

The runnable-but-blocked queue counts the number of processes that are otherwise runnable, but cannot be run because they are waiting for a block device transfer (generally a disk I/O operation).

The Linux vmstat looks slightly different, but displays the same information:

% vmstat 30

 procs                  memory    swap        io    system         cpu

 r b w  swpd  free  buff cache  si  so   bi   bo   in   cs  us  sy  id

 0 0 0 20068  9568 26524 50828   0   0    2    7    8    3   3   1   8

 0 0 0 20068  9568 26524 50828   0   0    0    0  115   22   8   4  89

 0 0 0 20068  9568 26524 50828   0   0    1    0  115   22   7   4  89

 0 0 0 20068  9568 26524 50828   0   0    0    0  114   20   7   4  89

3.6.3 Specific Breakdowns

Processor time is organized into four timed modes: system time, user time, I/O wait time, and idle time.[23] The idle time consists of what's left over when all other portions have had their fill. A program's normal operating state is user mode, but as it runs, it may generate requests for services that the kernel provides, such as I/O. These requests require the attention of the operating system, so the program switches into system mode, then returns to user mode when the request is complete. The time spent in these two modes is tabulated independently to give the user time and system time values, respectively. These two figures account for the majority of a process's execution time.

[23] Note that vmstat reports only the user time, system time, and idle time (wait time is summed in with idle time). In order to get separated values for wait time and idle time, use mpstat.

When a process waits for a block device data request to complete, it incurs I/O wait time. This brings up an important fact: when a process is blocked in this fashion, all idle time becomes wait time. If your idle time is zero, as reported by vmstat, the first thing you should check is if your system has I/O throughput problems.

Another good diagnostic event is looking at system time relative to user time. On NFS servers, which we will discuss in Section 7.5, it is normal for the system time to be much higher than the user time because NFS service is contained entirely inside the kernel.

You can obtain this sort of data on a per-process basis by means of the Solaris ptime command. Here is an example:

% /usr/proc/bin/ptime tar -cf miscellany.tar miscellany

real        7.402

user        0.044

sys         0.287

There are other commands, /bin/time and timex, which provide varying resolution. We'll discuss these commands in greater detail in Section 8.2.1.

There is a fair bit of system time incurred here because of the volume of I/O requests that must be made through the kernel facilities. There are many ways to get this sort of data on a systemwide level: vmstat and mpstat (which we'll discuss in more detail throughout this book) both provide it. The highlighted portion of the vmstat output is illustrative:

% vmstat 30

 procs     memory            page            disk          faults      cpu

 r b w   swap  free  re  mf pi po fr de sr s0 s1 -- --   in   sy  cs us sy id

 0 0 0   3296  2344   0 365 44 22 33  0  1  4  8  0  0  304  700  318 11  6 84

 0 0 0 1221816 99992  0 450  0  0  0  0  0  0  2  0  0  269 1791  465 19  6 76

 0 0 0 1222568 100688 0  83  0  0  0  0  0  0  8  0  0  254  591  389  2  1 98

This system exemplifies a system under light load. The idle percentage is fairly high and user time is greater than system time.

3.6.4 Multiprocessor Systems

The tools we've discussed present valid data for multiprocessor systems, but in order to get detailed per-processor statistics under Solaris, we need to turn to a more powerful tool: mpstat.

This is some sample mpstat output from a two-processor system:

% mpstat 30

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0  182   0  353   269   54  161   24   13    5    0   940   11   5   5  78

  1  182   1  531   134  116  157   23   13    6    0   931   10   6   5  79

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0    0   0 2856   228   28   72    0    6    0    0   454    2   2   4  92

  1    4   0    0    39   39   77    0    7    0    0    50    0   0   4  96

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0   80   0  515   489  288  108    2   11    1    0   275    1   6  37  56

  1   61   0   55    46   46  113    2    9    2    0   270    0   1  37  62

Here are some of the more interesting statistics:

  • intr is the number of interrupts handled by that processor; ithr is the number of interrupts as threads.

  • migr is the number of thread migrations. A thread migration occurs when a thread switches from running on one processor to running on some other processor.

  • smtx is the number of spins on mutexes; (in practical terms, this column represents the number of locks not acquired on the first try). We'll discuss this later in this chapter in Section 3.6.6.

  • syscl is the number of system calls.

  • usr, sys, wt, and idl are the user, system, wait, and idle percentages, on a per-processor basis. Note that vmstat provides one systemwide average only.

3.6.5 top and prstat

A popular and common tool for monitoring system-wide process activity is top, which provides a list of the most CPU-intensive processes running on a system, as well as some interesting statistics such as the load average, the number of processes, and the amount of memory and paging space in use. top is a very common and generally easy-to-understand application, so we won't discuss it further here. In the interest of completeness, this is what top output looks like:

% top

 10:27pm  up 24 days, 21:24,  9 users,  load average: 0.05, 0.06, 0.01

125 processes: 124 sleeping, 1 running, 0 zombie, 0 stopped

CPU states:  8.0% user,  9.1% system,  0.0% nice, 82.7% idle

Mem:  255596K av, 251056K used,   4540K free,      0K shrd,   3200K buff

Swap: 112444K av,      8K used, 112436K free                 99112K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND

 5752 gdm       17   0  1296 1296  1052 R       0 13.8  0.5   0:01 top

 3829 root       9   0  1252 1252  1072 S       0  1.1  0.4   0:06 sshd

 5753 root      10   0  1492 1492  1292 S       0  1.1  0.5   0:00 sendmail

    1 root       8   0   520  520   452 S       0  0.0  0.2   0:48 init

Unfortunately, the tradeoff for top's updating process list and other nice features is that it tends to consume quite a bit of processor time.[24] This has been known to substantially skew the behaviour of the system, so Solaris now ships with a similar tool, called prstat, that provides similar output. While it doesn't provide some of the nice information that top does about memory usage, it consumes very little processor time and doesn't skew the system's behaviour nearly as much:

[24] Rumor has it that this is where top got its name -- it often appears at the top of the CPU consumers list.

% prstat

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP

  4585 gdm      1280K 1096K cpu0    48    0   0:00.00 0.4% prstat/1

  4554 gdm      1424K 1024K sleep   58    0   0:00.00 0.0% csh/1

   337 nobody     23M 6856K sleep   59    0   0:00.07 0.0% ns-slapd/15

  4552 root     1784K 1256K sleep   38    0   0:00.00 0.0% in.telnetd/1

...

Total: 33 processes, 143 lwps, load averages: 0.00, 0.00, 0.01

The upshot of this is that top is very nice, but you need to be careful about it skewing the system under observation because of its high overhead. On Solaris systems, prstat provides similar output at a much reduced cost in terms of processor time.

3.6.6 Lock Statistics

In addition to using mpstat for processor state information, we can also use it to provide meaningful data on interrupts and mutexes. The key measurement is the smtx column, which represents the number of instances in the unit time that the kernel failed to immediately obtain a mutex when it asked for one. If this number begins to exceed 250 times the number of configured processors, system time will begin to increase noticeably. Higher performance processors can cope with higher smtx values, e.g., UltraSPARC-III/Pentium 3 Xeon.

% mpstat 30

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0  182   0  353   269   54  161   24   13    5    0   940   11   5   5  78

  1  182   1  531   134  116  157   23   13    6    0   930   10   6   5  79

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0  112   0  365   259   55  141    6   12    2    0   486    6   3  10  81

  1  157   0  337    73   67  147    7   11    3    0   594    6   4   8  82

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0   97   0  598   229   26  135    6   11    2    0   389    5   3   5  87

  1  107   0   81    67   62  130    5   11    2    0   423    4   3   4  89

The best way to cope with this problem is to check with the vendor for any operating system upgrades that may have solved mutex contention issues, and then use the lockstat command to find out what mutexes are suffering. If you are consistently seeing very high processor utilization and the smtx values are high in your SMP system, replace your current processors with faster ones, rather than adding more!

To find suffering mutexes, lockstat uses a kernel lock monitoring facility. It must be invoked as root. The kernel can dynamically change its locks to collect information, then return to the normal, optimized locking code when the data collection is complete. lockstat can be invoked on a per-process basis or on a system-wide basis by calling it on the sleep command. I went through some grief trying to find a simple example of how to cause a large number of mutex spins to obtain a good example for illustration, and finally did so:[25]

[25] I set this up by running a number of infinitely looping C shell scripts equal to the number of configured processors on the system. I hope this is not actually what your system is doing.

% mpstat 10

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0    1   0    2   104    0   19    0    0    0    0    26    0   0   0 100

  1    1   0    1     8    6   19    0    0    0    0    37    0   0   0 100

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0    0   0    0   109    2   40   22    2  956    0 106954   47  53   0   0

  1    0   0    0    23    7   43   15    2 1117    0 107760   48  52   0   0

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0    0   0    0   116    0   57   26    2 1289    0 107204   47  53   0   0

  1    0   0    0    24    3   45   21    2 1099    0 107199   47  53   0   0

It should be obvious that this system is having serious mutex problems. We can use lockstat to try and examine what the system is doing:

# lockstat sleep 10

Adaptive mutex spin: 19132 events

Count indv cuml rcnt     spin Lock                   Caller

-------------------------------------------------------------------------------

 2716  14%  14% 1.00        1 flock_lock             cleanlocks+0x1c

 2388  12%  27% 1.00        3 0xe0452ab0             dev_get_dev_info+0x3d

 1905  10%  37% 1.00        3 0xe0452ab0             cdev_size+0x48

 1554   8%  45% 1.00        1 0xe07c9f30             vn_rele+0x22

 1307   7%  52% 1.00        5 stable_lock            specvp+0x8f

 1050   5%  57% 1.00        2 0xe0452ab0             mod_rele_dev_by_major+0x21

  995   5%  62% 1.00        2 0xe0452ab0             ddi_hold_installed_driver+0x36

...

-------------------------------------------------------------------------------

...

Unfortunately, lockstat is quite verbose by default. However, at first glance it appears that the system is spending most of its time working with the flock_lock lock, which is filesystem-related. From this point it's a matter of using truss to see what system calls in our application are generating these contentions.

If you have a suspicion as to what lock might be the one causing the problem, lockstat has the capability to search for a particular lock. This is accomplished with the -l lockname switch:

# lockstat -l flock_lock sleep 10

Adaptive mutex spin: 2594 events

Count indv cuml rcnt     spin Lock                   Caller

-------------------------------------------------------------------------------

 2594 100% 100% 1.00        1 flock_lock             cleanlocks+0x1c

-------------------------------------------------------------------------------

Adaptive mutex block: 9 events

Count indv cuml rcnt     nsec Lock                   Caller

-------------------------------------------------------------------------------

    9 100% 100% 1.00 64490487 flock_lock             cleanlocks+0x1c

-------------------------------------------------------------------------------

3.6.7 Controlling Processors in Solaris

Solaris 2.6 and later versions support commands to help the system administrator manage processors. They are of somewhat limited use in most performance tuning applications, but they are good to know about.

3.6.7.1 psrinfo

The psrinfo command displays information concerning the configured processors. It provides particularly useful information when the -v switch is applied.

% /usr/sbin/psrinfo -v

Status of processor 0 as of: 09/20/99 16:24:34

  Processor has been on-line since 09/01/99 06:33:49.

  The sparc processor operates at 200 MHz,

        and has a sparc floating point processor.

Status of processor 1 as of: 09/20/99 16:24:34

  Processor has been on-line since 09/01/99 06:33:52.

  The sparc processor operates at 200 MHz,

        and has a sparc floating point processor.

This utility has proved to be useful when dealing with clients who were not sure what sort of hardware they had, but wanted to have it tuned. Similar information can be obtained by examining the contents of /proc/cpuinfo in Linux.

3.6.7.2 psradm

You can change the operational status of processors with the psradm command. A processor can have the following states:

  • An on-line processor does computation[26] and may be interrupted by the system's I/O devices.

    [26] That is, an on-line processor processes processes.

  • An off-line processor does not do any computation and is not usually interruptible. In some cases, it may not be possible to disable interrupts for an off-line processor.

  • A no-intr processor performs computation, but is not interruptible.

At least one processor must be able to perform computation, and at least one processor must be able to be interrupted.

3.6.7.3 psrset

It is sometimes useful to create processor sets through psrset. A processor set allows you to bind a process to a particular group of processors, rather than just one. Processor sets are sometimes created by the system, in which case they will run both system processes and user processes, or they can be created by psrset, in which case they are only able to run processes that have been bound to that processor. Let's walk through an example of using psrset on a Sun Ultra Enterprise 3500 system, and then binding a process to that processor set.

The system I use for this example has eight 250 MHz UltraSPARC-II processors:

# psrinfo

6       on-line   since 11/15/01 16:47:33

7       on-line   since 11/15/01 16:47:40

10      on-line   since 11/15/01 16:47:40

11      on-line   since 11/15/01 16:47:40

14      on-line   since 11/15/01 16:47:40

15      on-line   since 11/15/01 16:47:40

18      on-line   since 11/15/01 16:47:40

19      on-line   since 11/15/01 16:47:40

You can use psrset -i to see what the defined processor sets on a system are (for a listing by processor instead of by set, you can use psrset -p). By default, there are none -- all processes are in a general execution pool. Let's create a processor set with four processors (those numbered 14, 15, 18, and 19) using psrset -c:

# psrset -i

# psrset -c 14 15 18 19

created processor set 1

processor 14: was not assigned, now 1

processor 15: was not assigned, now 1

processor 18: was not assigned, now 1

processor 19: was not assigned, now 1

# psrset -i

user processor set 1: processors 14 15 18 19

# psrset -p

processor 14: 1

processor 15: 1

processor 18: 1

processor 19: 1

If you want to add more processors to this set, you can use psrset -a set-id processor-list; similarly, you can remove them with psrset -r processor-list.

# psrset -i

user processor set 1: processors 14 15 18 19

# psrset -a 1 10 11

processor 10: was not assigned, now 1

processor 11: was not assigned, now 1

# psrset -i

user processor set 1: processors 14 15 18 19 10 11

# psrset -r 18 19

processor 18: was 1, now not assigned

processor 19: was 1, now not assigned

# psrset -i

user processor set 1: processors 14 15 10 11

You can completely remove a processor set with psrset -d set-id:

# psrset -i

user processor set 1: processors 14 15 10 11

# psrset -d 1

removed processor set 1

# psrset -i

#

If you are configuring a mixed-use system -- for example, you might have to support both software development and production computational codes on the same system -- you can use processor sets to segment your workload. For example, on an eight-processor system, you might create one two-processor set dedicated for compiles, one four-processor set for computational jobs, and leave the other two processors unassigned (and therefore available for interactive use). One thing that you might want to do in that case is to disallow the processors that are dedicated to compilation and computation from handling interrupts (e.g., for disk or network I/O). This can be done by running psrset -f set-id; alternatively, you might want to set aside a few processors just for handling interrupts, in which case you can turn on interrupt handling for a set by using psrset -n set-id. By default, all processors are available to handle interrupts. For example, let's create a four-CPU processor set and disable interrupt handling on it:

# psrinfo

6       on-line   since 11/15/01 16:47:33

7       on-line   since 11/15/01 16:47:40

10      on-line   since 11/15/01 16:47:40

11      on-line   since 11/15/01 16:47:40

14      on-line   since 11/15/01 18:26:23

15      on-line   since 11/15/01 18:26:23

18      on-line   since 11/15/01 18:26:23

19      on-line   since 11/15/01 18:26:23

# psrset -i

# psrset -c 14 15 18 19

created processor set 1

processor 14: was not assigned, now 1

processor 15: was not assigned, now 1

processor 18: was not assigned, now 1

processor 19: was not assigned, now 1

# psrset -f 1

# psrinfo

6       on-line   since 11/15/01 16:47:33

7       on-line   since 11/15/01 16:47:40

10      on-line   since 11/15/01 16:47:40

11      on-line   since 11/15/01 16:47:40

14      no-intr   since 11/15/01 18:27:09

15      no-intr   since 11/15/01 18:27:09

18      no-intr   since 11/15/01 18:27:09

19      no-intr   since 11/15/01 18:27:09

Note that deleting a processor set that had interrupts disabled will not have the side effect of reenabling interrupts on its member processors. You can always reenable interrupts on a per-CPU basis with psradm -n processor-list.

Now that we've created a processor set, how can we run processes on it? There are essentially two ways: we can use psrset -e set-id command to run a command directly on the processor set, or we can use psrset -b pid-list to bind processes to a set.[27] For example, let's run a multithreaded application on processor set 1:

[27] Actually, it binds the underlying lightweight processes to the set (for more information on lightweight processes, see Section 3.3.2.1 earlier in this chapter).

# psrset -i

user processor set 1: processors 14 15 18 19

# psrset -e 1 mtsieve &

[1] 3701

# mpstat 3

...

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  6    0   0  747   401  301    0    0    0    0    0     0    0   0   0 100

  7    0   0    0   104  104   45    0    0    0    0    27    0   0   0 100

 10    0   0    0   100  100   13    0    0    0    0     9    0   0   0 100

 11    0   0    0   100  100   11    0    0    0    0     3    0   0   0 100

 14    0   0    0   111  100   11   11    0    0    0     0  100   0   0   0

 15    0   0    0   111  100   11   11    0    0    0     0  100   0   0   0

 18    0   0    0   111  100   11   11    0    0    0     0  100   0   0   0

 19    0   0    0   111  100   11   11    0    0    0     0  100   0   0   0

...

We can clearly see from the mpstat output that the CPUs in the processor set are completely saturated, but the other processors are totally idle.

3.6.8 Peripheral Interconnect Performance Tools

There are very few tools to monitor the performance of peripheral device interconnects. By far the best is the Solaris tool busstat, which lets you monitor the state of certain performance counters that have been embedded in Sun hardware. Because of the high degree of hardware support that is required, the capabilities of busstat may change over time, and from platform to platform. These examples were gathered on a Sun Ultra Enterprise 3500 running Solaris 8 7/01.

The best way to start using busstat is to see what devices are available for monitoring. This can be done via busstat -l:

# busstat -l

Busstat Device(s):

sbus0 sbus1 ac0 ac1 ac2 ac3 ac4

There are two sbus devices, which correspond to the two SBuses on this system's single SBus I/O card (see Section 3.5.1.4 earlier in this chapter). The ac devices correspond to the Address Controllers, which are on each installed board in the system. Each device has two performance instrumentation counters (PICs) that can count specific events. In order to get a list of the available events for each device, use busstat -e device:

# busstat -e sbus0

pic0

dvma_stream_rd

dvma_stream_wr

dvma_const_rd

dvma_const_wr

dvma_tlb_misses

dvma_stream_buf_mis

dvma_cycles

dvma_bytes_xfr

interrupts

upa_inter_nack

pio_reads

pio_writes

sbus_reruns

pio_cycles

pic1

...

Table 3-10 contains a description of the most important of these parameters.

Table 3-10. SBus performance counter events

Event

Description

dvma_stream_rd

The number of DVMA streaming reads issued

dvma_stream_wr

The number of DVMA streaming writes issued

dvma_const_rd

The number of DVMA consistent reads issued

dvma_const_wr

The number of DVMA consistent writes issued

dvma_tlb_misses

The number of transaction lookaside buffer (TLB) misses incurred during DVMA operations

dvma_stream_buf_mis

The number of DVMA streaming buffer read misses

dvma_cycles

The number of SBus cycles granted to DVMA operations

dvma_bytes_xfr

The number of bytes transferred across the SBus

interrupts

The number of interrupts generated by the SBus

upa_inter_nack

The number of nonacknowledged (on UPA) interrupts generated by the SBus

pio_reads

The number of programmed I/O reads issued

pio_writes

The number of programmed I/O writes issued

sbus_reruns

The number of SBus reruns

pio_cycles

The number of SBus cycles granted to PIO operations

And we can specify the counters that are to be examined by using busstat -w device,pic0=event,pic1=event. These counters will remain programmed until they are reset; the counters can only be programmed by the superuser.

These are obviously extremely detailed events, and may not be of great interest to anyone except driver developers or hardware engineers. However, one interesting thing to look at is the number of bytes transferred across the SBus. For example, we can use busstat to watch the activity on an SBus containing a network interface that is attached to a SunRay network appliance:[28]

[28] The SunRay is completely responsible for transmitting screen redrawing instructions on the host system. It functions somewhat like an X terminal.

# busstat -w sbus1,pic0=dvma_cycles,pic1=dvma_bytes_xfr 1

time dev    event0               pic0        event1               pic1

1    sbus1  dvma_cycles          1449        dvma_bytes_xfr       1940

2    sbus1  dvma_cycles          1361        dvma_bytes_xfr       1800

...

12   sbus1  dvma_cycles          1328        dvma_bytes_xfr       1800

13   sbus1  dvma_cycles          3469        dvma_bytes_xfr       6424

Up until this point, the system is simply sitting at the dtlogin screen. The small amount of data transfer is probably due to the "blinking" of the cursor. We then log in by typing a valid username and password:

14   sbus1  dvma_cycles          7282        dvma_bytes_xfr       19812

15   sbus1  dvma_cycles          7620        dvma_bytes_xfr       21172

16   sbus1  dvma_cycles          1399        dvma_bytes_xfr       2352

At this point we have successfully logged in, and the CDE "Welcome to Solaris" screen appears, followed by the user's desktop environment (twm and a few xterm processes):

17   sbus1  dvma_cycles          182874      dvma_bytes_xfr       590212

18   sbus1  dvma_cycles          1361        dvma_bytes_xfr       2004

19   sbus1  dvma_cycles          247957      dvma_bytes_xfr       814420

20   sbus1  dvma_cycles          2691        dvma_bytes_xfr       3688

21   sbus1  dvma_cycles          13542       dvma_bytes_xfr       25392

22   sbus1  dvma_cycles          7931        dvma_bytes_xfr       14248

After a little while, the SBus activity returns to near-zero levels, as there is only minimal graphical update activity occuring on the network:

31   sbus1  dvma_cycles          225         dvma_bytes_xfr       536

32   sbus1  dvma_cycles          597         dvma_bytes_xfr       760

Although this is a very limited example, it does show some of the powerful analysis made possible by busstat.

3.6.9 Advanced Processor Performance Statistics

The cpustat and cputrack tools in Solaris can be used to obtain extremely detailed statistics about processor performance. cpustat generates system-wide statistics, whereas cputrack lets you gather data about a specific process.

You can obtain the list of available events via cpustat -h. Some events are only available on specific PICs. Table 3-11 summarizes the various events, along with what PICs they are available on.

Table 3-11. CPU performance counter events

Event

Available PICs

Description

Cycle_cnt

0, 1

The number of accumulated clock cycles

Instr_cnt

0, 1

The number of completed instructions

Dispatch0_IC_Miss

0

The number of instruction cache misses

Dispatch0_mispred

1

The number of mispredicted branches (note that branch misprediction typically causes instructions in flight to be killed, so the number of pipeline bubbles is about twice as big as this count)

Dispatch0_storeBuf

0

The number of times that a store instruction was issued, but store buffer was full, and could not hold additional writes to memory

Dispatch0_FP_use

1

The number of times that an instruction depended on a floating-point result that was not yet available for reasons other than loading (that is, Dispatch0_FP_use and Load_use are mutually exclusive)

Load_use

0

The number of times that an instruction waiting to be executed required a load from memory that hadn't yet completed; stalls all instructions in the pipeline

Load_use_RAW

1

The number of times that a load-use hazard exists, but there is a read-after-write hazard on the oldest outstanding load. This indicates that load data is being delayed until an earlier store can be completed

IC_ref

0

The number of instruction cache references

IC_hit

1

The number of instruction cache hits

DC_rd

0

The number of data cache read references

DC_rd_hit

1

The number of data cache read hits

DC_wr

0

The number of data cache writes

DC_wr_hit

1

The number of data cache write hits

EC_ref

0

The number of E-Cache (L2 cache) references

EC_hit

1

The number of E-Cache hits

EC_write_hit_RD0

0

The number of E-Cache write hits that result in ownership being transferred

EC_wb

1

The number of E-Cache misses that incur writebacks to main memory

EC_snoop_inv

0

The number of E-Cache invalidations driven by bus snooping

EC_snoop_cb

1

The number of E-Cache copybacks driven by bus snooping

EC_rd_hit

0

The number of E-Cache read hits after a data cache miss

EC_ic_hit

1

The number of E-Cache read hits after an instruction cache miss

Again, some of these counters refer to very low-level performance structures, and will primarily be of use to people who are intimately familiar with the architecture of their processor. For more information about the UltraSPARC processor family architecture, consult the user guides published by Sun Microelectronics at http://www.sun.com/microelectronics/manuals/index.html.

One thing that is often interesting to compute is the number of cycles per instruction. For this example, I draw upon the simple Sieve of Eratosthenes application source code (see Section 8.2.4.2), with the minor modification that it will not display output (simply compute it). I run an experiment to compare the number of cycles per instruction required as a function of the compiler (Forte 6 Update 1) optimization level. Gather cputrack data for each process like this:

% cputrack -c pic0=Cycle_cnt,pic1=Instr_cnt sieve.o0

   time lwp      event      pic0      pic1

  1.046   1       tick 170075841 128955109

  2.056   1       tick 221955530 162910824

  3.056   1       tick 240994274  93274214

  4.056   1       tick 231824966  52868531

  5.056   1       tick 198733670  53220280

  6.056   1       tick  92304003  51971543

  6.522   1       exit 1192528514 564103611

We then compute, using the last line (the final statistics), the average number of cycles per instruction by dividing the pic0 entry (the number of cycles) by the pic1 entry (the number of instructions) to give a CPI of 2.114. As we increase the optimization, we get the results given here.

Compiler flags

Total instructions

Cycles per instruction

Total runtime (seconds)

-xO0

564,103,381

2.114

6.507

-xO2

168,642,699

5.022

5.480

-xO3

137,109,359

5.929

5.275

-fast -xO5

138,521,627

5.684

5.213

It's dangerous to overgeneralize these performance results, but it's interesting to watch the tremendous CPI jump as you move from no optimization to some optimization. Clearly, further optimization gives decreasing returns.

As we continue to further our understanding, we can use cputrack to try and answer some questions we might have about this application's runtime behavior. For example, we can compute the instruction cache hit rate for the unoptimized application:

%  ptime cputrack -c pic0=IC_ref,pic1=IC_hit sieve.o0

   time lwp      event      pic0      pic1

...

  6.475   1       exit 179350516 179325008

Therefore, we can compute that we have instruction cache hit rate of 99.986%, which is incredibly high. We can also determine the data cache read hit rate and data cache write hit rate:

% cputrack -c pic0=DC_rd,pic1=DC_rd_hit sieve.o0

   time lwp      event      pic0      pic1

...

  6.469   1       exit 128490011 128460760

real        6.509

user        0.038

sys         0.063

% cputrack -c pic0=DC_wr,pic1=DC_wr_hit sieve.o0

   time lwp      event      pic0      pic1

...

  6.471   1       exit  69216839  33654774

Finally, we end up with a data cache read hit rate of 99.977% and a write hit rate of 48.622%. At this point, things start to get interesting. We've been analyzing the unoptimized application, and our data cache write hit rate seems a little bit low. In fact, we might wonder whether it goes up as the optimization level increases. When we run that experiment, we find out that the write cache hit rate in the highest-optimized (and lowest-runtime) version has a write cache hit rate of only 0.031%. This seems confusing at first, but upon some consideration it makes sense: the application is performing at peak rates when it doesn't write often to the same memory location. This is a good illustration of how confusing examining these low-level counters can be without a great deal of knowledge about the application under analysis.


3.7 Concluding Thoughts

Microprocessors are some of the most complex devices per square centimeter that humanity has ever constructed, and understanding how they function is extremely challenging. I hope this chapter has given you some sense of that complexity, along with furthering your understanding and helping you prepare to measure a processor's performance in a dynamic system.

We've covered a lot, from the fundamentals of microprocessor design, to bus and crossbar architectures, to discussing caching and peripheral interconnects. This is a lot of information to grasp, but the payoff of understanding how these components function and interact with each other is commensurately large.


Chapter 4. Memory

Ideally one would desire an indefinitely large memory capacity such that any particular. . . word would be immediately available. . . .

—A. W. Burks, H. H. Goldstine, and J. von Neumann: Preliminary Discussion of the Logical Design of an Electronic Computing Instrument, 1946

Physical memory is a set of integrated circuits designed to store binary data. This storage has two characteristic properties: it is transient, as all stored information vanishes when electrical power is lost, and randomly accessible, meaning any bit can be accessed as fast as any other bit. In addition to physical memory, most systems implement virtual memory, which acts to manage physical memory and provide a simple interface to application developers. Virtual memory is consumed by the system kernel, filesystem caches, intimately shared memory, and processes.

Memory performance begins to affect overall system performance in two instances. The first instance occurs when the system is unable to retrieve and store data from physical memory fast enough, or when the system is forced to travel to main memory frequently. This sort of problem can be attacked by tuning the algorithm that is responsible or by buying a system with faster access to main memory. The second, and more likely, case is that the demand for physical memory by all currently running applications, including the kernel, exceeds the available amount. The system is then forced to begin paging , or writing unused pieces of memory to disk. If the low memory condition worsens, the memory consumed by entire processes will be written to disk, which is called swapping .[1] Memory conditions fall into four categories:

[1] Paging and swapping are often used interchangeably. However, they mean very different things in practice, so we'll be careful not to mix them.

  • Sufficient memory is available, and the system performs optimally.

  • Memory is constrained (one likely culprit, especially on older Solaris systems, is the filesystem cache). Performance begins to suffer as the system attempts to scavenge memory that is not in active use.

  • The system is legitimately short of memory. Performance suffers, especially on interactive processes.

  • Memory is critically scarce. Swapping begins to take place. System performance takes a heavy hit, and interactive performance becomes abysmal.

In this chapter, I'll describe how memory is physically implemented, the mechanics of how the system manages memory, and how paging and swapping function at a system level. I'll give you some tools you can use to analyze your memory use, and explain how to work with swap space. Finally, I'll cover mechanisms to address some common memory performance problems.


4.1 Implementations of Physical Memory

Let's start by looking at how memory is physically implemented in modern systems. All modern, fast memory implementations are accomplished via semiconductors,[2] of which there are two major types: dynamic random access memory (DRAM) and static random access memory (SRAM). The difference between them is how each memory cell is designed. Dynamic cells are charge-based, where each bit is represented by a charge stored in a tiny capacitor. The charge leaks away in a short period of time, so the memory must be continually refreshed to prevent data loss. The act of reading a bit also serves to drain the capacitor, so it's not possible to read that bit again until it has been refreshed. Static cells, however, are based on gates, and each bit is stored in four or six connected transistors. SRAM memories retain data as long as they have power; refreshing is not required. In general, DRAM is substantially cheaper and offers the highest densities of cells per chip; it is smaller, less power-intensive, and runs cooler. However, SRAM is as much as an order of magnitude faster, and therefore is used in high-performance environments. Interestingly, the Cray-1S supercomputer had a main memory constructed entirely from SRAM. The heat generated by the memory subsystem was the primary reason that system was liquid-cooled.

[2] In some cases, such as where resistance to ionizing radiation is required, magnetic core memory is still used.

There are two primary performance specifications for memory. The first represents the amount of time required to read or write a given location in memory, and is called the memory access time. The second, the memory cycle time, describes how frequently you can repeat a memory reference. They sound identical, but they are often quite different due to phenomena such as the need to refresh DRAM cells.

There is quite a gap between the speed of memory and the speed of microprocessors. In the early 1980s, the access time of commonly available DRAM was about 200 ns, which was shorter than the clock cycle of the commonly used 4.77 MHz (210 ns) microprocessors of the day. Fastforwarding two decades, the clock cycle time of the average home microprocessor is down to about a nanosecond (1 GHz), but memory access times are hovering around 50 ns.

Many different kinds of memory modules have been developed to improve system performance. I'll present a few here.

  • The oldest option currently in use is fast-page mode (FPM) memory. It implements the ability to read a full page (4 KB or 8 KB) of data during a single memory access cycle.

  • A refinement of this technique goes into extended data output (EDO) memory. The refinements are largely based on electrical modifications.

  • A more revolutionary change was implemented in synchronous DRAM (SDRAM) memory, which uses a clock to synchronize the input and output of signals. This clock is coordinated with the CPU clock, so the timings of all the components are synchronized. SDRAM also implements two memory banks on each module, which essentially doubles the memory throughput; it also allows multiple memory requests to be pending at once. A variation on SDRAM, called double-data rate SDRAM (DDR SDRAM) is able to read data on both the rising and falling edges of the clock, which doubles the data rate of the memory chip.

    SDRAM modules are usually described as PC66, PC100, or PC133, which refers to the clock rate they are driven at: 66 MHz (15 ns), 100 MHz (10 ns), or 133 MHz (8 ns), respectively. DDR SDRAM modules, however, are usually referred to by their peak throughput; a PC2100 module, for example, is theoretically capable of about 2.1 GB/sec peak throughput.

  • Direct Rambus (RDRAM) memory, however, takes an entirely different approach. It uses a narrow (16 bits wide) but extremely fast (600-800 MHz) path to memory, and allows sophisticated pipelining of operations. It is widely used in high-performance embedded applications (e.g., Sony's PlayStation 2 and the Nintendo64 consoles). RDRAM modules are usually called PC600, PC700, or PC800, which refers to their clock rate.

Workstations and servers tend to be able to interleave memory across banks, which is conceptually similar to disk striping (see Section 6.2.1); if we have 128 MB of memory configured as four 32 MB modules, we store one bit on each module in turn, rather than the first 32 MB on the first module, the second 32 MB on the second module, etc. This allows much higher performance because of the avoidance of cycle time delays. Interleaving is almost always done in power-of-two increments.[3]

[3] One exception is the Sun Ultra Enterprise hardware line, which will create 6-way interleaves.

It can also lead to confusion. Let's say we have a system with seven memory banks filled. The memory controller may well decide to make a four-way interleave, a two-way interleave, and a one-way interleave, which would mean that a memory-intensive process's performance would depend on where exactly it fell in memory. The best resource on how to configure your system's memory for optimum interleaving is the hardware vendor.


4.2 Virtual Memory Architecture

A virtual memory system exists to provide a framework for the system to manage memory on the behalf of various processes. The virtual memory system provides two primary benefits. It allows software developers to write to a simple memory model, which shields the programmer from the memory subsystem's hardware architecture and allows the use of memory sizes substantially greater than physical memory through backing stores. It also permits processes to have nonfragmented address spaces, regardless of how physical memory is organized or fragmented. In order to implement such a scheme, four key functions are required.

First, each process is presented with its own virtual address space; that is, it can "see" and potentially use a range of memory. This range of memory is equal to the maximum address size of the machine. For example, a process running on a 32-bit system will have a virtual address space of about 4 GB (232). The virtual memory system is responsible for managing the associations between the used portions of this virtual address space into physical memory.

Second, several processes might have substantial sharing between their address spaces. For example, let's say that two copies of the shell /bin/csh are running. Both copies will have separate virtual address spaces, each with their own copy of the executable itself, the libc library, and other shared resources. The virtual memory system transparently maps these shared segments to the same area of physical memory, so that multiple copies are not stored. This is analogous to making a hard link in a filesystem instead of duplicating the file.

However, sometimes physical memory will become insufficient to hold the used portions of all virtual address spaces. In this case, the virtual memory system selects less frequently used portions of memory and pushes them out to secondary storage (e.g., disk) in order to optimize the use of physical memory.

Finally, the virtual memory system plays one of the roles of an elementary school teacher: keeping the children in his care from interfering with each other's private things. Hardware facilities in the memory management unit perform this function by preventing a process from accessing memory outside its own address space.

4.2.1 Pages

Just like energy in quantum mechanics, memory is quantized; that is, it is organized into indivisible units. These units are called pages. The exact size of a page varies from system to system and is dependent on the processor's memory management unit (MMU) implementation. Large page sizes reduce MMU activity by reducing the number of page faults, and save kernel memory. However, they also waste some memory, since the system can only deal with memory in page-sized segments. A request for less than one full page of memory will be answered with the allocation of a full page. Typically, page sizes are 4 KB or 8 KB. On Solaris systems, you can find the page size via /usr/bin/pagesize or getpagesize(3C), while on Linux it is defined in the kernel header file asm/param.h as EXEC_PAGESIZE.

Page Sizes on Various Architectures

There are only three commonly encountered microprocessor designs that implement an 8 KB page size: the DEC Alpha, the original Sun SPARC processors (e.g., the Ross RT601/Cypress CY7C601/Texas Instruments TMS390C601A, which were used in the SPARCstation 2), and the Sun UltraSPARC designs. The Intel 80x86, the MIPS processors used in SGI systems, the Motorola/IBM PowerPC, and Sun's microSPARC and SuperSPARC series processors all use a 4 KB page size.

4.2.2 Segments

The pages that compose a process are grouped into several segments. Each process has at least four of these segments:

Executable text

Consists of the actual executable instructions in the binary. This is mapped from the on-disk binary and is read-only.

Executable data

Contains initialized variables present in the executable. This is mapped from the on-disk binary, but has permissions set to read/write/private (the private mapping ensures that changes to these variables during runtime are not reflected in the on-disk file, or in other processes sharing the same executable).

Heap space

Consists of memory allocated by means of malloc(3). This is called anonymous memory, because it has no mapping in the filesystem (we'll discuss it further in Section 4.3.2.1 later in this chapter).

Stack

Also allocated from anonymous memory.

4.2.3 Estimating Memory Requirements

Sometimes you have it easy, and a system only needs to run one commercial software package, for which the vendor has nicely told you precisely how much memory you need for optimal performance. Unfortunately, the real world is rarely so nice: the systems administrator in a typical computing environment has hundreds of different processes to worry about. Rather than try and solve every possible problem, we'll work out a general mechanism for thinking about how much memory is required for a given system.

The most important thing is to establish what a reasonable workload is on the system. Let's say that you have a system that's being used for serving typical low-end users: they run a shell, maybe an editor, a mailreader (probably pine or elm), or perhaps a newsreader. If you've been carefully monitoring your current environment, you'll be able to get a handle on how many of each sort of process is running at peak usage times. If you haven't, or if you're starting from scratch, you'll need to make a reasonable guess. Let's say that you've decided that there will be 50 users logged in at peak hours, and the usage pattern works out like this:

  • 5 invocations each of ksh and csh, 40 of tcsh

  • 25 pine processes and 10 elm processes

  • Other applications that defy categorization

We'll work through calculating the memory requirements of the csh processes. If we run pmap on a representative process, we get output that resembles this:

% pmap -x 7522

7522:   -csh

Address   Kbytes Resident Shared Private Permissions       Mapped File

0803C000      48      48       -      48 read/write/exec    [ stack ]

08048000     116     116     116       - read/exec         csh

08065000      16      16       -      16 read/write/exec   csh

08069000      72      72       -      72 read/write/exec    [ heap ]

DFF19000     548     444     408      36 read/exec         libc.so.1

DFFA2000      28      28       4      24 read/write/exec   libc.so.1

DFFA9000       4       4       -       4 read/write/exec    [ anon ]

DFFAB000       4       4       4       - read/exec

libmapmalloc.so.1

DFFAC000       8       8       -       8 read/write/exec

libmapmalloc.so.1

DFFAF000     148     136     136       - read/exec         libcurses.so.1

DFFD4000      28      28       -      28 read/write/exec   libcurses.so.1

DFFDB000      12       -       -       - read/write/exec    [ anon ]

DFFDF000       4       4       -       4 read/write/exec    [ anon ]

DFFE1000       4       4       4       - read/exec         libdl.so.1

DFFE3000     100     100     100       - read/exec         ld.so.1

DFFFC000      12      12       -      12 read/write/exec   ld.so.1

--------  ------  ------  ------  ------

total Kb    1152    1024     772     252

It looks like each process is taking about 1,150 KB of memory, with 1,024 KB resident in memory. Of that 1,024 KB, 772 KB is shared with other processes, and 252 KB is private. So, for five invocations, we could roughly estimate memory usage at roughly 2 MB (about 770 KB shared, plus five times about 250 KB). These are back-of-the-envelope calculations, but they are remarkably good at giving you an idea how much memory you need.

Remember, though, that memory is consumed by three other things (the filesystem cache, intimately shared memory, and the kernel) aside from processes! Unless you're running Oracle or another database application, you probably won't need to worry about intimately shared memory. As a rule of thumb, you can safely figure on about 32 MB for the kernel and other applications, plus another 16 MB if you're running a windowing system. If your users only access a few hundred megabytes of data, but access it frequently, it may be advisable to buy sufficient memory to cache the entire data set. This will drastically improve I/O performance to that information.

4.2.4 Address Space Layout

The exact details of a process's address space vary depending on the architecture it is implemented on. For Solaris systems, there are essentially four address space layouts:

  • SPARC V7 32-bit combined kernel and process address space (used on sun4c, sun4m, and sun4d architectures)

  • SPARC V9 32-bit separated kernel and process address space (used on sun4u machines)

  • SPARC V9 64-bit separated kernel and process address space (used on sun4u machines)

  • Intel IA-32 32-bit combined kernel and process address space (used on Solaris for Intel systems)

The SPARC V7 32-bit combined model maps the kernel's address space into the top of the process's address space. This means that the virtual address space available to the process is restricted by the amount consumed by the kernel's address space (256 MB on sun4c and sun4m architectures, and 512 MB on sun4d architectures). The kernel's address space is protected from the user's process by means of the the processor's privilege levels. The stack begins just beneath the kernel at address 0xEFFFC000 (0xDFFFE000 on sun4d systems) and grows downwards to address 0xEF7EA000 (0xDF7F9000 on sun4d), where libraries are loaded into memory. The text and data portions of the executable are loaded into the bottom of the address space, and the heap grows from the top of those upwards towards the libraries.

The UltraSPARC microprocessor architecture allows the kernel its own private address space, which removes the size limit on the kernel's address space.[4] Thus, for the 32-bit sun4u memory model, the stack begins at address 0xFFBEC000 and grows downwards until 0xFF3DC000 (the small space at the absolute top of the address space is reserved for the OpenBoot PROM). Otherwise, it is the same as the SPARC V7 models.

[4] This was a significant problem in the large pre-UltraSPARC systems, such as the SPARCcenter 2000E.

Although the UltraSPARC processor supports the SPARC V9 64-bit mode, the UltraSPARC-I and UltraSPARC-II processor implementations allow only 44-bit physical memory addresses. This creates a "hole" in the middle of the virtual address space, which spans addresses 0xFFFFF7FF.FFFFFFFF to 0x00000800.00000000.

The Intel architecture address space is similar to the sun4c/sun4m SPARC V7 model, in that is does not separate user space from kernel space. However, it has a significant difference: the stack is mapped beneath the executable segments (starting at address 0x8048000) and permitted to grow downwards to the very bottom of the address space.

4.2.5 The Free List

The free list is the mechanism by which the system dynamically manages the flow of memory between processes. Memory is taken from the free list by processes, and returned to it when the process exits or by the action of the page scanner, which tries to ensure that there is a small amount of memory free for immediate use at all times. Every time memory is requested, a page fault is incurred. There are three kinds of faults, which we'll discuss in more detail later in this section:

Minor page fault

Occurs every time a process needs some memory, or when a process tries to access a page that has been stolen by the page scanner but not recycled.

Major page fault

Occurs when a process tries to access a page that has been taken from it by the page scanner, completely recycled, and is now being used by another process. A major page fault is always preceded by a minor page fault.

Copy-on-write fault

Caused by a process trying to write to a page of memory that it shares with other processes.

Let's examine how the free list is managed under a Solaris system.

When a system boots, all of its memory is formed into pages, and a kernel data structure is created to hold their states. The kernel reserves a few megabytes of memory for itself and releases the rest to the free list. At some point, a process will request memory, and a minor page fault occurs. A page is then taken from the free list, zeroed, and made available to the process. This sort of behavior, in which memory is given out on an "as-needed" basis, has traditionally been called demand paging.

Pages are always taken from the head of the free list.

When the free list shrinks to a certain size (set to lotsfree in units of pages), the kernel wakes up the page scanner (also called the pagedaemon), which begins to search for pages that it can steal to replenish the free list. The page scanner implements a two-step algorithm order to avoid stealing pages that are being frequently accessed. The pagedaemon looks through memory in physical memory order and clears the MMU reference bit for each page. When a page is accessed, this bit is set. The page scanner then delays for a short while, waiting for pages to be accessed and their reference bits cleared. This delay is controlled by two parameters:

  • slowscan is the initial scan rate. Increasing this value causes the page scanner to run less unnecessary jobs, but do more work.

  • fastscan is the scan rate when the free list is completely empty.

The pagedaemon then goes through memory again. If a particular page still has the reference bit cleared, then the page hasn't been touched, so it is stolen for recycling. Think of memory as a circular track. A train rides on this track; as the engine in the front of the train passes a tie, that tie's reference bit is cleared. When the caboose gets around to a tie, if that tie's reference bit is still cleared, that page is recycled.

Some pages are very resistant to recycling, such as those that belong to the kernel and those that are shared between more than eight processes (this helps to keep shared libraries in memory). If the stolen page does not contain cached filesystem data, then it is moved to the page-out queue, marked as pending I/O, and eventually written out to swap space with a cluster of other pages. If the page is being used for filesystem caching, it is not written to disk. In either case, the page is placed on the end of the free list, but not cleared; the kernel remembers that this particular page still stores valid data.

If the average size of the free list over a 30-second interval is less than desfree, the kernel begins to take desperate measures to free memory: inactive processes are swapped out, and pages are written out immediately rather than being clustered. If the 5-second average of free memory is less than minfree, active processes start to be swapped out. The page scanner stops looking for memory to reclaim when the size of the free list increases above lotsfree.

The page scanner is governed by a parameter, maxpgio, which limits the rate at which I/O occurs to the paging devices. It is, by default, set very low (40 pages per second on sun4c, sun4m, and sun4u architectures, and 60 pages per second on sun4d), in order to prevent saturating the paging device with I/Os. This rate is inadequate for a modern system with fast disks, and should be set to 100 times the number of spindles of configured swap space. If a process tries to access a recycled page, another minor page fault is incurred. There are now two possibilities:

  • The page that was recycled has been placed back on the free list, but hasn't been reused yet. The kernel gives the page back to the process.

  • The page has been recycled completely, and is now in use by another process. A major page fault is incurred and the paged-out data is read into a new page, taken from the free list.

Since pages can be shared between multiple processes, special precautions need to be taken in order to ensure that changes to a page are maintained locally to that process. If a process tries to write to a shared page, it incurs a copy-on-write fault.[5] A page is taken from the free list, and a copy of the original shared page is made for the process. When a process completes, all of its nonshared pages are returned to the free list.

[5] Often called a COW fault; not to be confused with a ruminant falling into a chasm.

4.2.5.1 Virtual memory management in Linux

This mechanism is used in most modern memory management systems, but often with some minor changes. Here, I discuss the Linux 2.2 kernel, as at the time of this writing the 2.4 kernel was still undergoing significant revisions to the VM subsystem. The Linux kernel implements a different set of kernel variables that tune memory performance a little bit differently.

When the size of a Linux system's free list drops below the value of freepages.high, the system begins to gently page. When the available memory drops below the freepages.low value, the system pages heavily, and if the free list shrinks beneath freepages.min, only the kernel can allocate more memory. The freepages tunables are located in the /proc/sys/vm/freepages file, which is formatted as freepages.min freepages.low freepages.high.

The pagedaemon under Linux is called kswapd, which frees as many pages as necessary to get the system's free list back over the freepages.high mark. kswapd's behavior is controlled by three parameters, called tries_base, tries_min, and swap_cluster, located in that order in the /proc/sys/vm/kswapd file. The most important tunable is swap_cluster, which is the number of pages that kswapd will write out in a single turn. This value should be large enough so that kswapd does its I/O in large chunks, but small enough that it won't overload the disk I/O request queue. If you find yourself getting short on memory and paging heavily, you probably will want to experiment with increasing this value, to buy yourself more bandwidth to the paging space.

When a page fault is taken in Linux, multiple pages are actually read in to avoid multiple short trips to disk; the precise number of pages that are brought into memory is given by 2n, where n is the value of the page-cluster tunable (the only value in the /proc/sys/vm/page-cluster tunable file). It is pointless to set this value above 5.

There are two important things to take from the page lifecycle described here. The first is the manner by which the page scanner operates, and the parameters that define its behavior. The page scanner manages the ebb and flow of the free list by controlling paging, which can have a tremendous impact on performance. The second is the way pages are allocated and released; minor page faults occur when a process requests a page of memory, and major page faults occur if that requested page doesn't contain the data the process expects it to. Table 4-1 summarizes the kernel variables related to the behavior of the virtual memory system.

Table 4-1. A summary of memory-related kernel variables

Operating system

Variable

Effect

Solaris

lotsfree

The target size of the free list, in pages. Set by default, one sixty-fourth of physical memory, with a minimum size of 512 KB (64 or 128 pages on an 8 KB or 4 KB page size system, respectively).

The Linux equivalent is freemem/high.

desfree

The minimum pages of free memory, taken over an interval of 30 seconds, before swapping starts.

Set to one-half of lotsfree by default.

Roughly equivalent to Linux freemem/low.

minfree

The threshhold of minimum free memory, taken over a 5-second interval, before active swapping of processes starts. Represented in pages.

Set by default to one-half of desfree.

There is no equivalent parameter under Linux.

slowscan

The slowest scan rate (e.g., when scanning first starts) in pages.

Set by default to one sixty-fourth of the size of physical memory, with a minimum of 512 KB; that is, 64 pages on an 8 KB page size system, or 128 pages on a 4 KB page size system.

There is no Linux equivalent.

fastscan

The fastest scan rate (e.g., with the free list completely empty) in pages.

Set by default to half the physical memory, with a maximum of 64 MB; that is, 8,192 pages on an 8 KB page size system, or 16,384 pages on a 4 KB page size system.

There is no Linux equivalent.

maxpgio

The maximum number of I/Os per second that will be queued by the page scanner. Set by default to 40 on all systems except those based on the sun4d architecture, which has a default setting of 60.

There is no Linux equivalent.

Linux (2.2.x)

freepages.high

The target size of the free list in pages. When memory drops below this level, the system starts to gently page.

Set by default to 786.

The Solaris equivalent is lotsfree.

freepages.low

The threshhold for aggressive memory reclamation in pages.

By default, this is set to 512.

A rough Solaris equivalent is minfree.

freepages.min

The minimum number of pages on the free list. When this value is reached, only the kernel can allocate more memory.

Set to 256 by default.

There is no equivalent parameter under Solaris.

page-cluster

The number of pages that are read in on a page fault is given by 2page-cluster.

The default value is 4.

There is no equivalent parameter under Solaris.

4.2.6 Page Coloring

The precise way that pages are organized in processor caches can have a dramatic effect on application performance. The optimal placement of pages depends on how the application utilizes memory -- some applications access memory almost randomly, whereas others access memory in a strict sequential order -- so there is no single algorithm that provides universally optimal results.

The free list is actually organized into a number of specifically colored bins. The number of bins is given by the physical second-level cache size divided by the page size (for example, a system with a 64 KB L2 cache and an 8 KB page size would have 8 bins). When a page is returned to the free list, it is assigned to a specific bin. When a page is required, one is taken from a specific colored bin. That choice is made based on the virtual address of the requested page. (Recall that pages exist in physical memory, and are physically addressable, but processes only understand virtual addresses; a process incurs a page fault on a virtual address in its own address space, and that page fault is filled by a page with a physical address.) The algorithms we are discussing here decide how pages are assigned colors. There are three available page coloring algorithms in Solaris 7 and later:

  • The default algorithm (numbered 0) uses a hashing algorithm against the virtual address, in order to distribute pages as evenly as possible.

  • The first optional algorithm (numbered 1) assigns colors to pages so that physical addresses map directly to virtual addresses.

  • The second optional algorithm (numbered 2) assigns pages to bins in a round robin mechanism.

Another algorithm called Kessler's Best Bin (numbered 6) was supported on Ultra Enterprise 10000 systems running Solaris 2.5.1 and 2.6 only. It used a history-based mechanism to try and assign pages to the least-used bins.

The default paging algorithm was chosen because it consistently provides good performance. You are not likely to see any performance improvement from tuning these parameters on typical commercial workloads. However, scientific applications may see a significant performance gain -- some testing will reveal the page coloring algorithm that fits best with your application's memory usage patterns. You can change the algorithm by setting the system parameter consistent_coloring to the number of the algorithm you'd like to use.

4.2.7 Transaction Lookaside Buffers (TLB)

While we're discussing the physical to virtual address relationship, it's important to mention the transaction lookaside buffer (TLB). One of the jobs of the hardware memory management unit is to translate a virtual address, as provided by the operating system, into a physical address. It does this by looking up entries in a page translation table. The most recent translations are cached in the translation lookaside buffer. Intel and older SPARC processors (pre-UltraSPARC) use hardware mechanisms to populate the TLB, whereas the UltraSPARC architecture uses software algorithms. While a detailed discussion of the impact of the TLB on performance is a bit beyond the scope of this book, you should be generally aware of what a TLB is and what it does.


4.3 Paging and Swapping

Paging and swapping are terms that are often used interchangeably, but they are quite distinct. A system that is paging is writing selected, infrequently used pages of memory to disk, while a system that is swapping is writing entire processes from memory to disk. Let's say that you are working on your automobile, and you only have a small amount of space available for tools. Paging is equivalent to putting the 8 mm socket back in the toolchest so you have enough room for a pair of pliers; swapping is like putting your entire socket set away.

Many people feel that their systems should never nontrivially page (that is, perform paging on a pre-Solaris 8 system that is not simply filesystem activity). It's important to realize that paging and swapping allow the system to continue getting work done despite adverse memory conditions. Paging is not necessarily indicative of a problem; it is the action of the page scanner to try and increase the size of the free list by moving inactive pages to disk. A process, as a general rule, spends about 80% of its time running about 20% of its code; since the entire process doesn't need to be in memory at once, writing some pages out to disk won't affect performance substantially. Performance only begins to suffer when a memory shortage continues or worsens.

4.3.1 The Decline and Fall of Interactive Performance

Historically, Unix systems implemented a time-based swapping mechanism, whereby a process that was idle for more than 20 seconds would be swapped out; this isn't done anymore. Swapping is now used only to address the most severe memory shortages. If you come across a system where vmstat reports a nonzero swap queue, the only conclusion you can draw is that, at some indeterminate time in the past, the system was short enough on memory to swap out a process.

Memory shortages also tend to appear worse than they are because of the nature of the memory reclamation mechanism. When a system is desperately swapping jobs out, it tries to avoid a major performance decrease by keeping active jobs in memory for as long as possible. Unfortunately, programs that directly interact with users (such as shells, editors, or anything else that is dependent on user-supplied input) are inactive while waiting for the user to type something. As a result, these interactive processes are likely to be targeted as good places for memory reclamation efforts to be targeted. For example, when you pause momentarily in tying a command (such as when you're looking through the process table trying to find the ID of the process that is hogging all your memory so you can kill it), your shell will have to make the long trip from disk back into memory before your characters can be echoed back. Even worse, the disk subsystem is probably under heavy load from all the paging and swapping activity! The upshot is that in memory shortages, interactive performance falls through the floor.

This situation degrades even further if memory consumption continues. The system starts to slow down as disks become overloaded with paging and swapping, and the load average spirals higher; memory-based constraints quickly turn into I/O-based constraints.

4.3.2 Swap Space

Swap space (or, more accurately, paging space, since almost all of this sort of activity is paging rather than swapping) is a topic that tends to confuse people. Swap space serves three functions:

  • As a place to write pages of private memory (a paging store)

  • As a place for anonymous memory storage

  • As a place to store crash dumps[6]

    [6] By default, the very end of the first swap partition is used to store kernel crash dumps under Solaris 7.

This space is allocated from both spare physical memory and from a swap space on disk, be it a dedicated partition or a swapfile.

4.3.2.1 Anonymous memory

Swap space for anonymous memory is used in two stages. Anonymous memory reservations are taken out of disk-based swap, but allocations are taken out of physical memory-based swap. When anonymous memory is requested (via the malloc(3C) system call, for example), a reservation is made against swap, and a mapping is made against /dev/zero. Disk-based swap is used until none remains, in which case physical memory is used instead. Memory space that is mapped but never used stays reserved. This is common behavior in large database systems, and explains why applications such as Oracle require large amounts of disk-based swap, even though they are unlikely to allocate the entire reserved space.

When reserved pages are accessed for the first time, physical pages are taken from the free list, zeroed, and allocated rather than reserved. Should a page of anonymous memory be stolen by the page scanner, the data is written to disk-based swap space (that is, the allocation is moved from memory to disk), and the memory is freed for reuse.

4.3.2.2 Sizing swap space

There are many rules of thumb for how much swap space should be configured on a system, ranging from four times the amount of physical memory to half the amount of physical memory, and none of them are particularly good.

Under Solaris, one tool to get a long term picture of swap usage is /usr/sbin/swap -s. Here's an example from a workstation running Solaris 7 with 128 MB of physical memory and a 384 MB swap partition:

% swap -s

total: 12000k bytes allocated + 3512k reserved = 15512k used, 468904k available

Unfortunately, the names of these fields are misleading, if not downright incorrect. A more accurately labeled version would read:

total: 12000k bytes allocated + 3512k unallocated = 15512k reserved, 468904k available

When the available swap space reaches zero, the system will no longer be able to use more memory (until some is freed). From this report, it is obvious that the system has plenty of swap space configured.

You can also use /usr/sbin/swap -l to find out how much swap space is in use. Here's an example from the same system:

% swap -l

swapfile             dev  swaplo blocks   free

/dev/dsk/c1t1d0s1   32,129     16 788384 775136

In this case, there is a single swap device (a dedicated partition) with a total area of 688,384 512-byte blocks (about 384 MB). At the time this measurement was taken, virtually all of the swap space was free.

One tool you can use to monitor swap space usage over time is /usr/bin/sar -r interval, with interval specified in seconds:

% sar -r 3600

SunOS fermat 5.7 Generic sun4u    06/01/99

00:00:00 freemem freeswap

00:00:00     679   935313

01:00:00     680   937184

02:00:00     680   937184

sar reports the free memory in pages, and the free swap in disk blocks (really the amount of available swap space).

On Linux systems, you can get similar information from the file /proc/meminfo:

% cat /proc/meminfo | head -3 | grep -vi mem

        total:    used:    free:  shared: buffers:  cached:

Swap: 74657792 18825216 55832576

In general, if more than about half of your swap area is in use, you should consider increasing it. Disk is very cheap; skimping on swap space is only going to hurt you.

4.3.2.3 Organizing swap space

In order to minimize the impact of paging as much as possible, swap areas should be configured on the fastest possible disks. The swap partition should be on a low numbered slice (traditionally, slice one, with the root filesystem on slice zero); we'll explain why in Chapter 5. There is absolutely no reason, from a performance point of view, to place swap areas on slow disks, fast disks being accessed through slow controllers, fast disks that are already heavily loaded with I/O activity, or to have more than one swap area on a disk. The best strategy is to have a dedicated swapping area with multiple fast disks and controllers. If it is possible, try to put the swap area on a fast, lightly used disk. If it is not possible, however, try and put the swap file on the most heavily used partition. This minimizes the seek time to the disk; for a detailed discussion, see Section 5.4.10.1.

Solaris allows you to use remote files as swapping areas via NFS. This isn't a good way to achieve reasonable performance on modern systems. If you're configuring a diskless workstation, you have no choice unless you can survive without disk-based swap space. Measuring the level of I/O activity on a per-partition and per-disk basis is a topic that we'll cover in more detail in Section 5.5. The single most important issue in swap area placement is to put swap areas on low numbered partitions on fast, lightly used disks.

4.3.2.4 Swapfiles

Sometimes, you need to create a swap area in an "emergency" setting. The most effective way of doing this on a Solaris system is to use a swapfile, which is a file on disk that the system treats as swap space. In order to create one, you must first create an empty file with /usr/sbin/mkfile, then use /usr/bin/swap -a on the swapfile. Here's an example:

# mkfile 64m /swapfile

# /usr/bin/swap -a /swapfile

# /usr/bin/swap -l

swapfile             dev  swaplo blocks   free

/dev/dsk/c0t0d0s0   32,0      16 262944 232816

/swapfile              -      16  65520  65520

You may alternately specify the size of the swapfile to mkfile in blocks or kilobytes by specifying the units as b or k, respectively. You can remove the file from the swap space via /usr/bin/swap -d swapfile. Note that the swapfile will not be automatically activated at system boot.


4.4 Consumers of Memory

Memory is consumed by four things: the kernel, filesystem caches, processes, and intimately shared memory. When the system starts, it takes a small amount (generally less than 4 MB) of memory for itself. As it dynamically loads modules and requires additional memory, it claims pages from the free list. These pages are locked in physical memory, and cannot be paged out except in the most severe of memory shortages. Sometimes, on a system that is very short of memory, you can hear a pop from the speaker. This is actually the speaker being turned off as the audio device driver is being unloaded from the kernel. However, a module won't be unloaded if a process is actually using the device; otherwise, the disk driver could be paged out, causing difficulties. Occasionally, however, a system will experience a kernel memory allocation error. While there is a limit on the size of kernel memory,[7] the problem is caused by the kernel trying to get memory when the free list is completely exhausted. Since the kernel cannot always wait for memory to become available, this can cause operations to fail rather than be delayed. One of the subsystems that cannot wait for memory is the streams facility; if a large number of users try to log into a system at the same time, some logins may fail. Starting with Solaris 2.5.1, changes were made to expand the free list on large systems, which helps prevent the free list from ever being totally empty.

[7] This number is typically very large. On UltraSPARC-based Solaris systems, it is about 3.75 GB.

Processes have private memory to hold their stack space, heap, and data areas. The only way to see how much memory a process is actively using is to use /usr/proc/bin/pmap -x process-id, which is available in Solaris 2.6 and later releases.

Intimately shared memory is a technique for allowing the sharing of low-level kernel information about pages, rather than by sharing the memory pages themselves. This is a significant optimization in that it removes a great deal of redundant mapping information. It is of primary use in database applications such as Oracle, which benefit from having a very large shared memory cache. There are three special things worth noting about intimately shared memory. First, all the intimately shared memory is locked, and cannot ever be paged out. Second, the memory management structures that are usually created independently for each process are only created once, and shared between all processes. Third, the kernel tries to find large pieces of contiguous physical memory (4 MB) that can be used as large pages, which substantially reduces MMU overhead.

4.4.1 Filesystem Caching

The single largest consumer of memory is usually the filesystem-caching mechanism. In order for a process to read from or write to a file, the file needs to be buffered in memory. When this is happening, these pages are locked in memory. After the operation completes, the pages are unlocked and placed at the bottom of the free list. The kernel remembers the pages that store valid cached data. If the data is needed again, it is readily available in memory, which saves the system an expensive trip to disk. When a file is deleted or truncated, or if the kernel decides to stop caching a particular inode, any pages caching that data are placed at the head of the free list for immediate reuse. Most files, however, only become uncached upon the action of the page scanner. Data that has been modified in the memory caches is periodically written to data by fsflush on Solaris and bdflush on Linux, which we'll discuss a little later.

The amount of space used for this behavior is not tunable in Solaris; if you want to cache a large amount of filesystem data in memory, you simply need to buy a system with a lot of physical memory. Furthermore, since Solaris handles all its filesystem I/O by means of the paging mechanism, a large number of observed page-ins and page-outs is completely normal. In the Linux 2.2 kernel, this caching behavior is tunable: only a specific amount of memory is available for filesystem buffering. The min_percent variable controls the minimum percentage of system memory available for caching. The upper bound is not tunable. This variable can be found in the /proc/sys/vm/buffermem file. The format of that file is min_percent max_percent borrow_percent; note that max_percent and borrow_percent are not used.

4.4.2 Filesystem Cache Writes: fsflush and bdflush

Of course, the caching of files in memory is a huge performance boost; it often allows us to access main memory (a few hundred nanoseconds) when we would otherwise have to go all the way to disk (tens of milliseconds). Since the contents of a file can be operated upon in memory via the filesystem cache, it is important for data-reliability purposes to regularly write changed data to disk. Older Unix operating systems, like SunOS 4, would write the modified contents of memory to disk every 30 seconds. Solaris and Linux both implement a mechanism to spread this workload out, which is implemented by the fsflush and bdflush processes, respectively.

This mechanism can have substantial impacts on a system's performance. It also explains some unusual disk statistics.

4.4.2.1 Solaris: fsflush

The maximum age of any memory-resident modified page is set by the autoup variable, which is thirty seconds by default. It can be increased safely to several hundred seconds if necessary. Every tune_t_fsflushr seconds (by default, every five seconds), fsflush wakes up and checks a fraction of the total memory equal to tune_t_fsflushr divided by autoup (that is, by default, five-thirtieths, or one-sixth, of the system's total physical memory). It then flushes any modified entries it finds from the inode cache to disk; it can be disabled by setting doiflush to zero. The page-flushing mechanism can be totally disabled by setting dopageflush to zero, but this can have serious repercussions on data reliability in the event of a crash. Note that dopageflush and doiflush are complimentary, not mutually exclusive.

4.4.2.2 Linux: bdflush

Linux implements a slightly different mechanism, which is tuned via the values in the /proc/sys/vm/bdflush file. Unfortunately, the tunable behavior of the bdflush daemon has changed significantly from the 2.2 kernels to the 2.4 kernels. I discuss each in turn.

In Linux 2.2, if the percentage of the filesystem buffer cache that is "dirty" (that is, changed and needs to be flushed) exceeds bdflush.nfract, then bdflush wakes up. Setting this variable to a high value means that cache flushing can be delayed for quite a while, but it also means that when it does occur, a lot of disk I/O will happen at once. A lower value spreads out disk activity more evenly. bdflush will write out a number of buffer entries equal to bdflush.ndirty; a high value here causes sporadic, bursting I/O, but a small value can lead to a memory shortage, since bdflush isn't being woken up frequently enough. The system will wait for bdflush.age_buffer or bdflush.age_super, in hundredths of a second, before writing a dirty data block or dirty filesystem metadata block to disk. Here's a simple Perl script for displaying, in a pretty format, the values of the bdflush configuration file:

#!/usr/bin/perl

my ($nfract, $ndirty, $nrefill, $nref_dirt, $unused, $age_buffer, $age_super,

$unused, $unused) = split (/\s+/, `cat /proc/sys/vm/bdflush`, 9);

print "Current settings of bdflush kernel variables:\n";

print "nfract\t\t$nfract\tndirty\t\t$ndirty\tnrefill\t\t$nrefill\n\r";

print "nref_dirt\t$nref_dirt\tage_buffer\t$age_buffer\tage_super\t$age_super\n\r";

In Linux 2.4, about the only thing that didn't change was the fact that bdflush still wakes up if the percentage of the filesystem buffer cache that is dirty exceeds bdflush.nfract. The default value of bdflush.nfract (the first in the file) is 30%; the range is from 0 to 100%. The minimum interval between wakeups and flushes is determined by the bdflush.interval parameter (the fifth in the file), which is expressed in clock ticks.[8] The default value is 5 seconds; the minimum is 0 and the maximum is 600. The bdflush.age_buffer tunable (the sixth in the file) governs the maximum amount of time, in clock ticks, that the kernel will wait before flushing a dirty buffer to disk. The default value is 30 seconds, the minimum is 1 second, and the maximum is 6,000 seconds. The final parameter, bdflush.nfract_sync (the seventh in the file), governs the percentage of the buffer cache that must be dirty before bdflush will activate synchronously; in other words, it is the hard limit after which bdflush will force buffers to disk. The default is 60%. Here's a script to extract values for these bdflush parameters in Linux 2.4:

[8] There are typically 100 clock ticks per second.

#!/usr/bin/perl

my ($nfract, $unused, $unused, $unused, $interval, $age_buffer, $nfract_sync, $u

nused, $unused) = split (/\s+/, `cat /proc/sys/vm/bdflush`, 9);

print "Current settings of bdflush kernel variables:\n";

print "nfract $nfract\tinterval $interval\tage_buffer $age_buffer\tnfract_sync $

nfract_sync\n";

If the system has a very large amount of physical memory, fsflush and bdflush (we'll refer to them generically as flushing daemons) will have a lot of work to do every time they are woken up. However, most files that would have been written out by the flushing daemon have already closed by the time they're marked for flushing. Furthermore, writes over NFS are always performed synchronously, so the flushing daemon isn't required. In cases where the system is performing lots of I/O but not using direct I/O or synchronous writes, the performance of the flushing daemons becomes important. A general rule for Solaris systems is that if fsflush has consumed more than five percent of the system's cumulative nonidle processor time, autoup should be increased.

4.4.3 Interactions Between the Filesystem Cache and Memory

Because Solaris has an untunable filesystem caching mechanism, it can encounter problems under some specific instances. The source of the problem is that the kernel allows the filesystem cache to grow to the point where it begins to steal memory pages from user applications. This behavior not only shortchanges other potential consumers of memory, but it means that the filesystem performance becomes dominated by the rate at which the virtual memory subsystem can free memory.

There are two solutions to this problem: priority paging and the cyclic cache.

4.4.3.1 Priority paging

In order to address this issue, Sun introduced a new paging algorithm in Solaris 7, called priority paging, which places a boundary around the filesystem cache.[9] A new kernel variable, cachefree, is created, which scales with minfree, desfree, and lotsfree. The system attempts to keep cachefree pages of memory available, but frees filesystem cache pages only when the size of the free list is between cachefree and lotsfree.

[9] The algorithm was later backported to Solaris 2.6.

The effect is generally excellent. Desktop systems and on-line transaction processing (OLTP) environments tend to feel much more responsive, and much of the swap device activity is eliminated; computational codes that do a great deal of filesystem writing may see as much as a 300% performance increase. By default, priority paging has been disabled until sufficient end-user feedback on its performance is gathered. It will likely become the new algorithm in future Solaris releases. In order to use this new mechanism, you need Solaris 7, or 2.6 with kernel patch 105181-09. To enable the algorithm, set the priority_paging variable to 1. You can also implement the change on a live 32-bit system by setting the cachefree tunable to twice the value of lotsfree.

4.4.3.2 Cyclic caching

A more technically elegant solution to this problem has been implemented in Solaris 8, primarily due to efforts by Richard McDougall, a senior engineer at Sun Microsystems. No special procedures need be followed to enable it. At the heart of this mechanism is a straightforward rule: nondirty pages that are not mapped anywhere should be on the free list. This rule means that the free list now contains all the filesystem cache pages, which has far-reaching consequences:

  • Application startup (or other heavy memory consumption in a short period of time) can occur much faster, because the page scanner is not required to wake up and free memory.

  • Filesystem I/O has very little impact on other applications on the system.

  • Paging activity is reduced to zero and the page scanner is idle when sufficient memory is enabled.

As a result, analyzing a Solaris 8 system for a memory shortage is simple: if the page scanner reclaims any pages at all, there is a memory shortage. The mere activity of the page scanner means that memory is tight.

4.4.4 Interactions Between the Filesystem Cache and Disk

When data is being pushed out from memory to disk via fsflush, Solaris will try to gather modified pages that are adjacent to each other on disk, so that they can be written out in one continuous piece. This is governed by the maxphys kernel parameter. Set this parameter to a reasonably large value (1,048,576 is a good choice; it is the largest value that makes sense for a modern UFS filesystem). As we'll discuss in Section 6.6 in Chapter 6, a maxphys value of 1,048,576 with a 64 KB interlace size is sufficient to drive a 16-disk RAID 0 array to nearly full speed with a single file.

There is another case where memory and disk interact to give suboptimal performance. If your applications are constantly writing and rewriting files that are cached in memory, the in-memory filesystem cache is very effective. Unfortunately, the filesystem flushing process is regularly attempting to purge data out to disk, which may not be a good thing. For example, if your working set is 40 GB and fits entirely in available memory, with the default autoup value of 30, fsflush is attempting to synchronize up to 40 GB of data to disk every 30 seconds. Most disk subsystems cannot sustain 1.3 GB/second, which will mean that the application is throttled and waiting for disk I/O to complete, despite the fact that all of the working set is in memory!

There are three telltale signs for this case:

  • vmstat -p shows very low filesystem activity.

  • iostat -xtc shows constant disk write activity.

  • The application has a high wait time for file operations.

Increasing autoup (to, say, 840) and tune_t_fsflushr (to 120) will decrease the amount of data sent to disk, improving the chances of issuing a single larger I/O (rather than many smaller I/Os). You will also improve your chances of seeing write cancellation, when not every modification to a file is written to disk. The flip-side is that you run a higher risk of losing data in the case of server failure.


4.5 Tools for Memory Performance Analysis

Tools for memory performance analysis can be classed under three basic issues: how fast is memory, how constrained is memory in a given system, and how much memory does a specific process consume? In this section, I examine some tools for approaching each of these questions.

4.5.1 Memory Benchmarking

In general, monitoring memory performance is a function of monitoring memory restraints. Tools for providing benchmarks as to how fast the memory subsystem is do exist; they are largely of academic interest, as it is unlikely that much tuning will be able to increase these numbers. The one exception to this rule is that users can carefully tune interleaving as appropriate. Most systems handle interleaving by purely physical means, so you may have to purchase additional memory: consult your system hardware manual for more information.[10] Nonetheless, it is often important to be aware of relative memory subsystem performance in order to make meaningful comparisons.

[10] Keep in mind that having not enough slightly faster memory is probably worse than having enough slightly slower memory. Be careful.

4.5.1.1 STREAM

The STREAM tool is simple; it measures the time required to copy regions of memory. This measures "real-world" sustainable bandwidth, not the theoretical "peak bandwidth" that most computer vendors provide. It was developed by John McCalpin while he was a professor at the University of Delaware.

The benchmark itself is easy to run in single-processor mode (the multiprocessor mode is quite a bit more complex; consult the benchmark documentation for current details). Here's an example from an Ultra 2 Model 2200:

$ ./stream

-------------------------------------------------------------

This system uses 8 bytes per DOUBLE PRECISION word.

-------------------------------------------------------------

Array size = 1000000, Offset = 0

Total memory required = 22.9 MB.

Each test is run 10 times, but only

the *best* time for each is used.

-------------------------------------------------------------

Your clock granularity/precision appears to be 1 microseconds.

Each test below will take on the order of 40803 microseconds.

   (= 40803 clock ticks)

Increase the size of the arrays if this shows that

you are not getting at least 20 clock ticks per test.

-------------------------------------------------------------

WARNING -- The above is only a rough guideline.

For best results, please be sure you know the

precision of your system timer.

-------------------------------------------------------------

Function      Rate (MB/s)   RMS time     Min time     Max time

Copy:         226.1804       0.0709       0.0707       0.0716

Scale:        227.6123       0.0704       0.0703       0.0705

Add:          276.5741       0.0869       0.0868       0.0871

Triad:        239.6189       0.1003       0.1002       0.1007

The benchmarks obtained correspond to the summary in Table 4-2.

Table 4-2. STREAM benchmark types

Benchmark

Operation

Bytes per iteration

Copy

a[i] = b[i]

16

Scale

a[i] = q * b[i]

16

Add

a[i] = b[i] + c[i]

24

Triad

a[i] = b[i] + q * c[i]

24

It is also interesting to note that there are at least three ways, all in common use, of counting how much data is transferred in a single operation:

Hardware method

Counts how many bytes are physically transferred, as the hardware may move a different number of bytes than the user specified. This is because of cache behavior; when a store operation misses in a cache, many systems perform a write allocate, which means that the cache line containing that memory location is stored in the processor cache.[11]

[11] This is done in order to ensure coherency between main memory and the caches.

Bcopy method

Counts how many bytes get moved from one location in memory to another. If it takes your machine one second to read a certain number of bytes at one location and one more second to write the same number of bytes to another location, the resulting bandwidth is that number of bytes per second.

STREAM method

Counts how many bytes the user asked to be read and how many bytes the user asked to be written. For the simple "copy" test, this is precisely twice the number obtained by the bcopy method. This is done because some of the tests perform arithmetic; it makes sense to count both the data read into and the data written back from the CPU.

One of the nice things about STREAM is that it uses the same method of counting bytes, all the time, so it's safe to make comparisons.

STREAM is available in source code, so that it can be easily compiled. A list of benchmarks are also available. The STREAM home page is located at http://www.streambench.org.

4.5.1.2 lmbench

Another tool for measuring memory performance is lmbench. While lmbench is capable of many sorts of measurements, we'll focus on four specific ones. The first three measure bandwidth: the speed of memory reads, the speed of memory writes, and the speed of memory copies (via the bcopy method described previously). The last one is a measure of memory read latency. Let's briefly discuss what each of these benchmarks measures:

Memory copy benchmark

Allocates a large chunk of memory, fills it with zeroes, and then measures the time required to copy the first half of the chunk of memory into the second half. The results are reported in megabytes moved per second.

Memory read benchmark

Allocates a chunk of memory, fills it with zeroes, and then measures the time required to read that memory as a series of integer loads and adds; each 4-byte integer is loaded and added to an accumulator variable.

Memory write benchmark

Allocates a chunk of memory, fills it with zeroes, and then measures the time needed to write that memory as a series of 4-byte integer stores and increments.

Memory latency benchmark

Measures the time required to read a byte from memory. The results are reported in nanoseconds per load. The entire data memory hierarchy[12] is evaluated, including the latency to the primary caches, secondary caches, main memory, and the latency effects of a TLB miss. In order to derive the most information from the lmbench memory latency benchmarks, try plotting the data as a function of the latency versus the array size used for the test.

[12] Most notably, any separate instruction caches are not measured.

When you compile and run the benchmark, it will ask you a series of questions regarding what tests you would like to run. The suite is fairly well documented. The lmbench home page, which contains source code and more details about how the benchmark works, is located at http://www.bitmover.com/lm/lmbench/.

4.5.2 Examining Memory Usage System-Wide

It is important to understand a system's performance. The only way to get that understanding is through regular monitoring of data.

4.5.2.1 vmstat

vmstat is one of the most ubiquitous performance tools. There is one cardinal rule about vmstat, which you must learn and never forget: vmstat tries to present an average since boot on the first line. The first line of vmstat output is utter garbage and must be discarded.

With that in mind, Example 4-1 shows output under Solaris.

Example 4-1. vmstate output in Solaris
# vmstat 5

 procs     memory             page            disk         faults       cpu

 r b w  swap  free    re  mf pi po fr de sr s0 s1 s2 --  in   sy   cs  us sy id

 0 0 0  43248 49592   0   1  5  0  0  0  0  0  1  0  0  116  106   30  1  1  99

 0 0 0 275144 56936   0   1  0  0  0  0  0  2  0  0  0  120    5   19  0  1  99

 0 0 0 275144 56936   0   0  0  0  0  0  0  0  0  0  0  104    8   19  0  0 100

 0 0 0 275144 56936   0   0  0  0  0  0  0  0  0  0  0  103    9   20  0  0 100

The r, b, and w columns represent the number of processes that have been in the run queue, blocked for I/O resources (including paging), and processes that are runnable but swapped, respectively. If you ever see a system with a nonzero number in the w field, all that you can infer is that the system was, at some point in the past, low enough on memory to force swapping to occur. Here's the most important data you can glean from vmstat output:

swap

The amount of available swap space, in KB.

free

The amount of free memory (that is, the size of the free list) in KB. In Solaris 8, this number includes the amount of memory used for the filesystem cache; in prior releases, it does not, and therefore will be very small.

re

The number of pages reclaimed from the free list. A page had been stolen from a process, but then reclaimed by the process before it was recycled and given to another process.

mf

The number of minor page faults. These are fast as long as a page is available from the free list, because they can be resolved without performing a page-in.

pi, po

Page-ins and page-outs, respectively, in KB per second.

de

The short term memory deficit. If the value is nonzero, memory has been vanishing fast lately, and extra free memory will be reclaimed with the expectation that it'll be needed soon.

sr

The scan rate activity of the pagedaemon, in pages per second. This is the most critical memory-shortage indicator. In Solaris systems prior to 8, if this stays about 200 pages per second for a reasonably long period of time, you need more memory. If you are running Solaris 8 and this number is nonzero, you are short of memory.

On Linux, the output is a little bit different:

% vmstat 5

 procs                  memory    swap        io    system         cpu

 r b w  swpd  free  buff cache  si  so   bi   bo   in   cs  us  sy  id

 1 0 0 18372  8088 21828 56704   0   0    1    7   13    6   2   1   6

 0 0 0 18368  7900 21828 56708   1   0    0    8  119   42   6   3  91

 1 0 0 18368  7880 21828 56708   0   0    0   14  122   44   6   3  91

 0 0 0 18368  7880 21828 56708   0   0    0    5  113   24   2   2  96

 0 0 0 18368  7876 21828 56708   0   0    0    4  110   27   2   2  97

Note that while the w field is calculated, Linux never desperation-swaps. swpd, free, buff, and cache represented the amount of virtual memory used, the amount of idle memory, the amount of memory used as buffers, and the amount used as cache memory, respectively. There is generally very little to be gathered from a Linux vmstat output. The most important thing to watch is probably the si and so columns, which indicate the amount of swap-ins and swap-outs. If these grow large, you probably need to increase the kswapd swap_cluster kernel variable to buy yourself some additional bandwidth to and from swap, or purchase more physical memory (see Section 4.2.5.1 earlier in this chapter).

4.5.2.2 sar

sar (for system activity reporter) is, like vmstat, an almost ubiquitous performance-monitoring tool. It is particularly useful in that it can be adapted to gather data on its own for later perusal, as well as doing more focused, short-term data collection. In general, its output is comparable with vmstat's, although differently labeled.

In general, the syntax for invoking sar is sar -flags interval number. This causes a specific number of data points to be gathered every interval seconds. When looking at memory statistics, the most important flags are -g, -p, and -r. Here's an example of the output generated:

$ sar -gpr 5 100

SunOS islington.london-below.net 5.8 Generic_108528-03 sun4u    02/19/01

11:28:10  pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf

          atch/s  pgin/s ppgin/s  pflt/s  vflt/s slock/s

         freemem freeswap

...

11:28:50     0.00     0.00     0.00     0.00     0.00

          251.60    5.00    5.20 1148.20 2634.60    0.00

           86319  3304835

The most important output fields are summarized in Table 4-3.

Table 4-3. sar memory statistics fields

Flag

Field

Meaning

-g

pgout/s

Page-out requests per second

ppgout/s

Pages paged out per second

pgfree/s

Pages placed on the free list per second by the page scanner

pgscan/s

Pages scanned per second by the page scanner

%ufs_ipf

The percentage of cached filesystem pages taken off the free list while they still contained valid data; these pages are flushed and cannot be reclaimed (see Section 5.4.2.7)

-p

atch/s

Page faults per second that are satisfied by reclaiming a page from the free list (this is sometimes called an attach)

pgin/s

The number of page-in requests per second

ppgin/s

The number of pages paged in per second

pflt/s

The number of page faults caused by protection errors (illegal access to page, copy-on-write faults) per second

-r

freemem

The average amount of free memory

freeswap

The number of disk blocks available in paging space

4.5.2.3 memstat

Starting in the Solaris 7 kernel, Sun implemented new memory statistics to help evaluate system behavior. To access these statistics, you need a tool called memstat. While currently unsupported, it will hopefully be rolled into a Solaris release soon. This tool provides some wonderful functionality, and is available at the time of this writing from http://www.sun.com/sun-on-net/performance.html. Example 4-2 shows the sort of information it gives you.

Example 4-2. memstat
# memstat 5

memory ---------- paging----------executable- -anonymous---filesys  --  ---  cpu --

  free  re  mf  pi  po  fr de sr epi epo epf  api apo apf  fpi fpo fpf us sy wt  id

 49584   0   1   5   0   0  0  0   0   0   0    0   0   0    5   0   0  1  1  1  98

 56944   0   1   0   0   0  0  0   0   0   0    0   0   0    0   0   0  0  0  0 100

Much like vmstat, the first line of output is worthless and should be discarded. memstat breaks page-in, page-out, and page-free activity into three different categories: executable, anonymous, and file operations. Systems with plenty of memory should see very low activity in the epf and apo fields. Consistent activity in these fields indicates a memory shortage.

If you are running Solaris 7 or earlier and do not have priority paging enabled, however, executables and anonymous memory will be paged with even the smallest amount of filesystem I/O, once memory falls to or beneath the lotsfree level.

4.5.3 Examining Memory Usage of Processes

It's nice to know how much memory a specific process is consuming. Each process's address space, which is made up of many segments, can be measured in the following ways:

  • The total size of the process address space (represented by SZ or SIZE)

  • The resident size (the size of the part of the address space that's held in memory) of the process address space (RSS)

  • The total shared address space

  • The total private address space

There are many good tools for examining memory usage and estimating the amount of memory required for a certain process.

4.5.3.1 Solaris tools

One common tool is /usr/ucb/ps uax. Here's an example of the sort of data it gives:

% /usr/ucb/ps uax

USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND

root     16755  0.1  1.0 1448 1208 pts/0    O 17:33:35  0:00 /usr/ucb/ps uax

root         3  0.1  0.0    0    0 ?        S   May 24  6:19 fsflush

root         1  0.1  0.6 2232  680 ?        S   May 24  3:10 /etc/init -

root       167  0.1  1.3 3288 1536 ?        S   May 24  1:04 /usr/sbin/syslogd

root         0  0.0  0.0    0    0 ?        T   May 24  0:16 sched

root         2  0.0  0.0    0    0 ?        S   May 24  0:00 pageout

gdm      14485  0.0  0.9 1424 1088 pts/0    S 16:17:57  0:00 -csh

Note that kernel daemons like fsflush, sched, and pageout report SZ and RSS values of zero; they run entirely in kernel space, and don't consume memory that could be used for running other applications. However, ps doesn't tell you anything about the amount of private and shared memory required by the process, which is what you really want to know.

The newer and much more informative way to determine a process's memory use under Solaris is to use /usr/proc/bin/pmap -x process-id. (In Solaris 8, this command moved to /usr/bin.) Continuing from the previous example, here's a csh process:

% pmap -x 14485

14485:  -csh

Address   Kbytes Resident Shared Private Permissions       Mapped File

00010000     144     144       8     136 read/exec         csh

00042000      16      16       -      16 read/write/exec   csh

00046000     136     112       -     112 read/write/exec    [ heap ]

FF200000     648     608     536      72 read/exec         libc.so.1

FF2B0000      40      40       -      40 read/write/exec   libc.so.1

FF300000      16      16      16       - read/exec         libc_psr.so.1

FF320000       8       8       -       8 read/exec         libmapmalloc.so.1

FF330000       8       8       -       8 read/write/exec   libmapmalloc.so.1

FF340000     168     136       -     136 read/exec         libcurses.so.1

FF378000      40      40       -      40 read/write/exec   libcurses.so.1

FF382000       8       -       -       - read/write/exec    [ anon ]

FF390000       8       8       8       - read/exec         libdl.so.1

FF3A0000       8       8       -       8 read/write/exec    [ anon ]

FF3B0000     120     120     120       - read/exec         ld.so.1

FF3DC000       8       8       -       8 read/write/exec   ld.so.1

FFBE4000      48      48       -      48 read/write         [ stack ]

--------  ------  ------  ------  ------

total Kb    1424    1320     688     632

This gives a breakdown of each segment, as well as the process totals.

4.5.4 Linux Tools

The ps command under Linux works essentially the same way it does under Solaris. Here's an example (I've added the header line for ease of identifying the individual columns):

% ps uax | grep gdm

USER       PID %CPU %MEM  SIZE   RSS TTY STAT START   TIME COMMAND

gdm      12329  0.0  0.7  1548   984  p3 S   18:18   0:00 -csh

gdm      13406  0.0  0.3   856   512  p3 R   18:37   0:00 ps uax

gdm      13407  0.0  0.2   844   344  p3 S   18:37   0:00 grep gdm

While ps is ubiquitous, it's not very informative. The Linux equivalent of Solaris's pmap are the entries in the /proc filesystem, which tell you very useful things.

One of the nicest features in Linux is the ability to work with the /proc filesystem directly. This access gives you good data on the memory use of a specific process, akin to pmap under Solaris. Every process has a directory under its process ID in the /proc filesystem; inside that directory is a file called status. Here's an example of what it contains:

% cat /proc/12329/status

Name:   csh

State:  S (sleeping)

Pid:    12329

PPid:   12327

Uid:    563     563     563     563

Gid:    538     538     538     538

Groups: 538 100

VmSize:     1548 kB

VmLck:         0 kB

VmRSS:       984 kB

VmData:      296 kB

VmStk:        40 kB

VmExe:       244 kB

VmLib:       744 kB

SigPnd: 0000000000000000

SigBlk: 0000000000000002

SigIgn: 0000000000384004

SigCgt: 0000000009812003

CapInh: 00000000fffffeff

CapPrm: 0000000000000000

CapEff: 0000000000000000

As you can see, this is a lot of data. Here's a quick Perl script to siphon out the most useful parts:

#!/usr/bin/perl

$pid = $ARGV[0];

@statusArray = split (/\s+/, `grep Vm /proc/$pid/status`);

print "Status of process $pid\n\r";

print "Process total size:\t$statusArray[1]\tKB\n\r";

print "            Locked:\t$statusArray[4]\tKB\n\r";

print "      Resident set:\t$statusArray[7]\tKB\n\r";

print "              Data:\t$statusArray[10]\tKB\n\r";

print "             Stack:\t$statusArray[13]\tKB\n\r";

print " Executable (text):\t$statusArray[16]\tKB\n\r";

print "  Shared libraries:\t$statusArray[19]\tKB\n\r";


4.6 Concluding Thoughts

There's a simple moral to this story: your memory requirements depend heavily on how many users there are and what they are doing. Large technical applications such as computational fluid dynamics or protein structure calculations can require incredible amounts of memory. It is easy to imagine a computer with 2 GB of memory that could support only one such job, whereas that same system could support several hundred interactive users quite comfortably.

One of the greatest improvements in computing in the last two decades has been the ability to assemble large amounts of memory for relatively little cost. Take advantage of it.


5.1 Disk Architecture

A disk drive consists of a number of stacked metal disks coated with a magnetic recording medium; these disks are called platters. The platters rotate around a central spindle, generally at speeds between 5,400 and 15,000 rotations per minute. Between these platters are a set of movable arms, which have small sensors called read/write heads. In order for a disk operation to complete to a given part of the disk, the arms must move into place, and then the appropriate location on the platter must spin beneath the heads. With proper positioning thusly achieved, data can now be read from, or written to the disk.[1]

[1] The disk drive also stores some vital information, like the disk label, at the very beginning of the disk.

The usable storage area of the disk is divided up based on an address scheme.[2] The ring of data that is swept out by the heads on a given platter as the platter rotates is called a track; the group of tracks on all platters that share a given arm position is called a cylinder. Cylinders are numbered from the outside edge of the disk inward; that is, the lowest numbered cylinders are also the outermost. Each track is further broken up into sectors, which are generally 512 bytes[3] long. The number of sectors per track is dependent on the size of the platter and the recording density. Figure 5-1 is an illustration of disk architecture.

[2] Some space is reserved for the disk drive's internal use; it stores things like head alignment data and error correction data.

[3] This size is mandated by the SCSI standard.

Figure 5-1. Disk architecture

5.1.1 Zoned Bit Rate (ZBR) recording

If the number of sectors per track were constant, data would become less dense across tracks as you moved outward along the disk, wasting space as you move outward. To address this problem, all modern disk drives use a technique called zone bit rate (ZBR) recording, in which the disk is divided up into zones, and more sectors are stored on the outer tracks.[4] Lower-numbered zones have more sectors per track than higher-numbered zones. This results in a roughly constant data density, in terms of bits per inch; it also causes the data rate to vary with the cylinder number.

[4] Most modern disks have about fifteen zones.

The lowest numbered cylinder is the fastest, largest, and most dense (most bits/cylinder); the highest numbered cylinder is the slowest, smallest, and least dense (fewest bits/cylinder).

Unfortunately, this makes disk performance calculations somewhat complicated, since the transfer rate varies depending on which cylinder you're accessing.

5.1.1.1 Disk caches

SCSI and IDE disks typically have local caches associated with them. These caches are simply a small amount of memory, typically between 128 KB and 4 MB, associated with the disk's embedded controller. When data is read from the drive, a copy is stored in this cache. Because most of the delay in accessing a disk is either in the time spent transferring data from the platters or in the time spent moving the read/write heads into position and waiting for the appropriate sector to fly by, servicing a data request from the disk's local cache saves a lot of time, and for commonly accessed data, the local caches can be very effective. By default, most drives have read caching enabled.

What is usually not enabled is the ability to use this local cache memory to cache writes. Typically, a drive will not acknowledge a write as complete until the data is physically written to disk. If write caching is enabled, the disk will acknowledge the write as complete when the data is stored in this memory. This memory is volatile, which means that its contents vanish when power is lost: this is not a problem for caching reads, but a power failure at an inopportune moment could cause an acknowledged write to be silently discarded because the data hasn't been physically written to disk. There is absolutely no guarantee that anything written to the disk buffer cache will actually ever see the platters. Because of this reliability issue, these caches are not normally enabled. Enabling them can substantially increase performance; however, this performance improvement comes with the risk of data loss.

We'll discuss how to enable and check the status of these caches in Section 5.5.1 later in this chapter.

5.1.2 Access Patterns

Disk accesses tend to be either clustered together (sequential access) or spread out in no particular order (random access). Just because disk accesses are clustered together does not necessarily mean that they are sequential; most disk accesses that happen to be close together on the disk end up being sequential. Clustered accesses aren't necessarily, but often are, accesses to sequential sectors on the disk. Unfortunately, these terms also have relevance in other areas: it is possible for media to be "randomly accessible" in the sense that "any bit can be accessed about as fast as any other bit" (such as a hard drive), or "sequentially accessible" (such as a tape drive). When I use these terms in this chapter, I only refer to the pattern of accesses.

The performance of the two access patterns is given by two very different rules:

  • Sequential access is dominated by the actual transfer time from the disk platters (internal transfer speed).

  • Random access is dominated by the platter rotational speed and the seek time.

Increasing the transfer size or transforming random access patterns into sequential access patterns significantly increases disk subsystem performance. This is one of the most profitable areas to target for improvements in performance.

5.1.3 Reads

A disk read is normally synchronous; the requesting process blocks until the read is complete. This is not always the case, as an asynchronous read could have been issued by aioread(3). When the system reads the data, it is cached in case it is needed again. Caching uses the main memory area and is described in detail in Section 4.4.1. If you are debating expanding main memory or investing in a big disk controller in the interest of improving read performance, you are almost always better off expanding main memory; you will have an expanded filesystem cache that is accessible in less than a microsecond, and you will almost certainly reduce the number of reads issued, saving on bus loading and CPU time.

Generally, the goals in designing a read-oriented disk subsystem are to minimize the latency for the initial read of uncached data and to maximize throughput for sequential data.

5.1.4 Writes

Disk writes are a mix of synchronous and asynchronous events. Asynchronous events occur when a file is written to the local filesystem: the data is written to memory and the write is immediately acknowledged. This data is actually flushed out to disk later by means of the filesystem flush daemon (see Section 4.4.2). Applications can request that a file be written out immediately by calling fsync(3C) or by closing the file. Note that when a process exits, all of its open filehandles are closed and therefore flushed. This can take a while and cause a substantial amount of disk activity if many processes exit at the same time.

5.1.4.1 UFS write throttling

In order to prevent too much memory from being consumed by pending writes, Solaris implements a write throttle algorithm. This algorithm limits the amount of data pending to any given file; it is a per-process, per-file limitation. For each file, between 256 KB (the low-water mark, set by the ufs:ufs_LW variable) and 384 KB (the high-water mark, set by the ufs:ufs_HW variable) of data can be pending at any one time. When there is less data than the low-water mark pending to a file, flushing the data is left to the filesystem flush daemon. When the amount of pending data is between the two water marks, writes are scheduled to push the data out to disk. If more than the high-water mark data is pending, the application is not allowed to write any more data (that is, the write( ) system call will block) until the amount of pending data drops below the low-water mark again.

At high-data rates, asynchronous writes to disk become synchronous, and processes slow down. The writes are asynchronous because they are being cached in memory and immediately acknowledged, and the disk access happens later. When the process's filesystem write buffer is full for a specific file, writes become synchronous, because the limit on performance is now how fast data can be written to the disk, rather than how fast data can be cached in memory.

When you want to write quickly to a file and the underlying disk subsystem can cope with the throughput, it may still be impossible to drive the disk subsystem to maximum efficiency because the write throttle kicks in too early. The write throttle can also affect applications that perform large bursts of write activity at spaced intervals. The best diagnostic for this condition is the ufs_throttles kernel counter. It is incremented every time the write throttle kicks in, so a steadily increasing value means that you should bump up the write throttle.

Increasing the write throttle has the side effect of queuing a larger number of I/O requests for a file; this method can improve throughput, but it increases the average response time, since the queue is longer. A simple way to see the effect of changing the write throttle is to use ptime(1) to look at the time required to create a file with mkfile. Please note that you probably do not want to use /tmp for this test. The reason is described in Section 5.4.8 later in this chapter.

In practice, the watermarks are set far too low by default. The high-water mark should be set to many times the value of the maxphys tunable (for a description of this variable, see Section 4.4.4), and the low-water mark should be set to one-half to two-thirds of the high-water mark. In practice, I often set the low-water mark equal to 1/32nd of physical memory, and the high-water mark is set to 1/16th of physical memory. This means that if you have, say, one process writing full-out to sixteen separate files, you are in theory able to use all of the system's memory as a write cache. However, in practice, you would never exhaust memory this way, because the pagedaemon would kick in and start reaping cached pages and pushing them out to disk. For more information on the pagedaemon, see Section 4.3.

The entire write throttle mechanism is controlled by the ufs:ufs_WRITES kernel tunable; by default it is set to one, which corresponds to enabling the throttle. If you set it to zero, it is turned off. The potential side effect of this is very high memory consumption.

5.1.5 Performance Specifications

One of the problems in analyzing disk performance is the confusing way most vendors provide performance statistics for their devices.

5.1.5.1 One million bytes is a megabyte?

Disk vendors, as well as the SCSI specification, use the term megabyte to mean one million (1,000,000) bytes. The convention everywhere else, however, is to use megabyte to mean 1,024 kilobytes, or 220 (1,048,576) bytes. This five-percent difference applies to both performance specifications and storage specifications:

  • A disk drive rated to transfer 8.4 million bytes per second can only transfer 8.0 MB per second.

  • A disk drive with an advertised capacity of 9,100 million bytes will only store 8.67 GB.

This convention is adhered to by the entire industry, despite the confusion it causes.

5.1.5.2 Burst speed versus internal transfer speed

Disk vendors also like to quote the burst speed of the embedded controller. This specifies the rate that the embedded controller can transmit data on the bus. It does not account for command overhead or state changes. It is simply the speed that bits are transmitted, which can only be sustained if the disk can provide data at that speed and the target can deal with a flow that fast. Practically, this speed only happens when data is being transferred from the disk's local cache to the host system's memory. In most cases, the data does not reside in the disk's cache, and so the operation must be satisfied from the disk platters. For random I/O, the disk probably also needs to move the disk arm into position.

The internal transfer speed reflects the speed that bits are read from the disk platters to the read/write head, exclusive of all other factors (such as seeking). This measures only how fast data can be pulled off the platters; it doesn't measure the performance of any other part of the disk drive. The internal transfer speed is much lower than the burst speed, and can vary substantially depending on which cylinder is being read (see Section 5.1.1 earlier in this chapter). As a result, the internal transfer speed is a reasonably good predictor of sequential transfer performance.

The most important consequence of the difference between burst speed and internal transfer speed is that a single disk with a 40 MB per second burst speed is unlikely to be any faster than an otherwise identical disk with a 20 MB per second burst speed.

5.1.5.3 Internal transfer speed versus actual speed

Let's further complicate matters.

The internal transfer speed reflects the true performance abilities of a disk drive more than burst speed does. However, the internal transfer speed measures only the speed at which all data flies past the read/write heads, and this measurement includes a substantial amount of overhead. Much of this overhead is spent on error-correcting codes that prevent most disk errors from resulting in corrupted data. By applying the following formula, we can get a more accurate measure of a single disk's performance:

Here's an example, using the Seagate 18.2 GB Barracuda disk (ST118273W). By consulting the manufacturer's specification sheet to find the average number of sectors per track (237) and the rotational speed of the drive (7,200 rpm), we can perform the following computation:

This represents the speed that data can be transferred to/from the disk, assuming purely sequential access.

5.1.5.4 Average seek time

The average seek time is another number that is often quoted, but it is remarkably misleading. According to the disk vendors, the average seek time is the sum of all possible seeks over the number of possible seeks. While this is mathematically true, it turns out that the physics involved in moving the disk arm mean that short seeks take much less time than long seeks. In fact, the "average" seek does not at all resemble a "typical" seek -- it involves moving the disk arm much farther than in a typical seek.

The lesson here is that storing data in a way that minimizes seek time optimizes performance. Splitting a single disk into two slices, each of which is busy, turns out to be a very bad idea. In such a configuration, every time the disk fields an access for the slice it's not currently on, it will need to seek from the average position in one slice to the average position in the other slice, which is half the maximum seek time. This takes a long time. It would be a better idea to put the two slices on different disks, in order to minimize the seek time.

It is easy to disregard the effects of excessive seek distance. However, with maximum seek times in the neighborhood of 20 ms, every maximum-length seek incurs a delay equal to hundreds of millions of instructions.

5.1.5.5 Storage capacity and access capacity

For random-access activity, disk performance is dominated by the platter rotational speed and the seek time. Consequently, drives that spin at the same rate perform about the same, largely irrespective of capacity or physical size.[5] For random-access activity, the most effective way to increase access capacity is to buy as many fast-spinning drives as possible!

[5] 5.25" drives have slightly higher seek times, but this is not likely to have any practical impact.

Unfortunately, increases in access capacity (generally quantified as random access operations per second) have not kept pace with increases in storage capacity. To illustrate this principle, let's consider a 5,400 rpm 2.1 GB disk and a 7,200 rpm 9.1 GB disk. The larger disk has more than four times the storage capacity, but only about 30% greater access capacity. Therefore, the greatest random access I/O capacity is usually achieved by using several of the smallest feasible disks, rather than one larger disk with superior specifications! This is illustrated in Table 5-1.

Table 5-1. Access capacity comparisons for various 218 GB installations

Disks[6]

Number

Buses[7]

I/O ops per second per disk[8]

Aggregate I/O ops per second

Aggregatetransfer rate

4.3 GB

51

4

108

5,508

160 MB/sec

9.1 GB

24

2

85

2,040

80 MB/sec

18.2 GB

12

1

70

840

40 MB/sec

[6] All disk types rotate at the same speed.

[7] The minimum number of Ultra Wide (fast-40) SCSI buses required to support the drives.

[8] These numbers are empirical.

However, using smaller disks is usually more expensive, more complicated, and takes more space. It also necessitates the use of disk arrays, which we'll discuss later, to solve the problems brought about by increased subsystem failure rate that is observed when more components are used. In most environments, it is preferable to design mixtures of large and small capacity disks to ride the "sweet spot" of the disk cost/performance curve as much as possible.

Configuring disk subsystems based only on storage considerations is generally a serious mistake from a performance point of view.


5.2 Interfaces

An interface serves as the rules governing the communication between the host computer system and a set of devices. These rules are usually established through the auspices of a host adapter, which resides on one of the system's peripheral interconnects (see Section 3.5).

The modern disk market is largely driven by three disk protocols: IDE, SCSI, and Fibre Channel. We'll focus primarily on these protocols, as well as one older protocol, IPI, and some of the emerging storage interfaces (IEEE 1394/Firewire and USB). It is worth pointing out that there are no "native" IEEE 1394 or USB devices; they are all IDE drives with adapters attached, and as such have many of the same problems as a strictly IDE device does.

5.2.1 IDE

IDE, for Integrated Drive Electronics, or alternatively, ATA, for AT Attachment, have historically been used in low-cost systems, particularly Intel-based personal computers. However, IDE disks are starting to become common in low-end workstations, such as the Sun Ultra 5 and Ultra 10, in order to keep prices as low as possible. The IDE specification was intended to reduce interface costs by placing the controller hardware on the drive itself, as well as making firmware implementations as easy as possible.

IDE development is managed to some degree by a technical committee known as T13, which is part of the National Committee on Information Technology Standards (NCITS, pronounced "insights"). NCITS is chartered by and functions under rules that are approved by the American National Standards Institute (ANSI). These rules try to ensure that voluntary standards are developed by industry group consensus. NCITS develops Information Processing System standards, while ANSI approves the process under which they are developed and publishes them. The T13 Technical Committee web page is located at http://www.t13.org.

Following the success of the original ATA interface, the Small Form Factor Committee (an organization of drive manufacturers) created a backwards-compatible extension of the ATA interface called ATA-2 (alternatively Fast-ATA). This standard added faster PIO and DMA modes. Contrary to popular suspicion, the logical block addressing (LBA) scheme that was revamped in ATA-2 had nothing to do with surpassing the 504 MB barrier; even the original ATA specification had a capacity limit in excess of 100 GB.

Several years later, a further revision of the specification called ATA-3 had been released, which added more sophisticated power management, increased reliability, and added failure prediction. It did not define any faster modes. One of the major disadvantages of ATA is that it was designed for disks only; with the explosive popularity of CD-ROM drives, the interface was extended by means of the ATA Packet Interface (ATAPI), which facilitates adding such devices. ATAPI was formally integrated into ATA with the ATA-4 standard.

Enhanced IDE (EIDE) is a marketing term referring to ATA-2 and ATAPI implementations, whereas Fast-ATA is a competing marketing term that builds on ATA-2 only.

One of the consequences of the low-cost goals of the IDE device is that the host processor ends up taking a substantial role in processing I/O requests. IDE drives have essentially two ways to transfer data to the host processor: programmed I/O (PIO), and direct memory access, or DMA, which can be performed on a single-word or multi-word level. A PIO transfer mode causes the host processor to handle every transaction from disk into memory itself, whereas in a DMA transfer mode the IDE controller can write directly into the host's memory. The drive communicates its ready status to the controller by asserting the IRQ line of the drive, which can be done in two cases:

  • A read command has been issued to the drive; that command has been serviced and the requesting data is in the drive's buffer, ready to be taken up and transferred into memory.

  • A write command has been issued to the drive; that command has been transferred to the drive's buffer (if the drive does not have write caching enabled, the line will not be asserted until the data has been physically committed to disk).

During normal reads and writes, handling this interrupt can continuously bother the CPU and cause long delays in servicing the drive. The Read/Write Multiple commands are drive-level commands that attempt to address this problem by allowing up to 128 sectors to be transferred without incurring any intervening interrupts, substantially lessening the overhead on the host processor.

Single-word DMA performs very poorly relative to multi-word DMA; in fact, "DMA" usually carries with it the implicit assumption of multi-word transfers. Single-word DMA support was dropped in the ATA-3 specification.

As hard disks continued to get faster, the fastest multi-word DMA mode (at 16.7 MB/s) quickly became insufficient. Unfortunately, the interface was really designed for slow data transfer (~5 MB/s). Simply increasing the clock rate of the interface caused all sorts of signaling interference problems; as a result, a new type of DMA transfer mode (called UltraDMA) was introduced in a standard called Ultra-ATA, which was developed to bridge the gap between ATA-3 and the then-forthcoming ATA-4. Now that ATA-4 has been released, the Ultra-ATA standard is integrated into ATA-4.

The key technological advance introduced in UltraDMA was double-transition clocking: data is transferred on both the rising and falling edges of the clock signal. This is exactly the same technology used to improve transfer rates in DDR SDRAM (see Section 4.1). In order to improve data integrity, UltraDMA modules also use a cyclical redundancy checking (CRC) algorithm to compute an error-detection value for each block of data transferred, which is sent along with the block. This allows the recipient of the data to determine whether the data was corrupted in transit, in which case the data is retransmitted.

Despite the throughput increase given by double-transition clocking, transferring data faster than 33 MB/s finally exhausted the capabilities of the standard 40-conductor IDE cable. To use UltraDMA modes over 2, a special 80-conductor cable is required.[9] This cable uses the same 40 pins, but adds 40 ground lines between each of the original 40 signal lines in order to separate those lines and prevent interference.

[9] The cable was specified in the ATA-4 specification for UltraDMA modes 0, 1, and 2, but was optional.

These standards are reviewed in Table 5-2.

Table 5-2. IDE interfaces

Type

Mode

Cycle time (MHz)[10]

Data rate (MB/sec)

Supported in

PIO

0

1.67

3.3

ATA

1

2.61

5.2

ATA

2

4.17

8.3

ATA

3

5.56

11.1

ATA-2

4

8.33

16.6

ATA-2

5

11.11

22.2

Vaporware!

DMA (single-word)

0

1.04

2.1

ATA

(Removed in ATA-3)

1

2.08

4.2

ATA

(Removed in ATA-3)

2

4.17

8.3

ATA

(Removed in ATA-3)

DMA (multi-word)

0

2.08

4.2

ATA

1

6.67

13.3

ATA-2

2

8.33

16.6

ATA-2

UltraDMA

0

4.17

16.6

ATA-4

1

6.25

25.0

ATA-4

2

(UDMA/33)

8.33

33.3

ATA-4

3

11.11

44.4

ATA-5

4

(UDMA/66)

16.67

66.7

ATA-5

5

(UDMA/100)

25

100.0

ATA-6?

[10] ATA cycle times are usually represented in nanoseconds, but for the ease of comparison, they are shown here in MHz.

Table 5-3 provides a summary of the IDE implementations and the features it supports.

Table 5-3. Characteristics of IDE implementation

EIDE

Fast-ATA

Fast-ATA-2

Ultra-ATA

PIO Transfer Modes

Up to 3

Up to 3

Up to 4

All

DMA (Single-word) Modes

All

All

All

All

DMA (Multi-word) Modes

Up to 1

Up to 1

Up to 2

All

Read/Write Multiple

No

Yes

Yes

Yes

Data Checksumming

No

No

No

Yes

The implementations described here are supported in Linux kernels after 2.2.0, and support for the higher-performance Ultra-ATA can be patched into older kernels. The Linux IDE kernel driver, which was written by Mark Lord, has a particularly low overhead for setting up transactions; it is excellent for handing small data transfers.

5.2.1.1 Improving IDE performance in Linux

There are a few things the Linux kernel doesn't do by default that can increase IDE I/O performance, such as telling the IDE device driver to use 32-bit I/O, and ensuring that the disk is performing DMA transfers. In the 2.2.16 kernel, these are all off by default. This is best accomplished via hdparm (see Section 5.5.2.1 later in this chapter).

To see whether your disk is using 32-bit IDE transfers, use the -c switch:

# hdparm -c /dev/hda

/dev/hda:

I/O support = 0 (default 16-bit)

To enable 32-bit transfers, use -c 1:

# hdparm -c 1 /dev/hda

/dev/hda:

setting 32-bit I/O support flag to 1

I/O support = 1 (32-bit)

Similarly, you can check the status of the IDE DMA transfer mode via -d:

# hdparm -d /dev/hda

/dev/hda:

using_dma = 0 (off)

You can enable IDE DMA by specifying -d 1:

# hdparm -d 1 /dev/hda

/dev/hda:

setting using_dma to 1 (on)

using_dma = 1 (on)

You may also see some performance increase by tweaking the number of sectors transferred per interrupt (set by the -m switch to hdparm). As always, benchmark the changes to make sure that you're actually improving performance. When you've found settings you'd like to keep, run hdparm -k 1 /dev/hd X to save your settings across an IDE reset.

5.2.1.2 Limitations of IDE drives

IDE is sufficient for small applications that are not very performance-sensitive, particularly if they do not issue many concurrent disk I/O operations and the overall disk subsystem load is relatively light.

IDE is an attractive, low-cost option; at the time of this writing (mid-2001), commodity IDE drives are approximately half the price and twice the capacity of equivalent-performance SCSI drives. Unfortunately, IDE has several limitations; chief amongst these are its very limited capacity for expansion per channel and the fact that it is entirely single-threaded on a per channel basis (command overlap is not allowed, even with two disks on the channel). In comparison, SCSI offers flexible device attachment, support for almost any peripheral type, and command overlap. Higher-performance devices also tend to be available for SCSI well in advance of their availability for IDE.

5.2.2 IPI

When the SCSI standard was defined, a second mass-storage standard aimed at high-performance, high-cost installations was defined, known as the Intelligent Peripheral Interface (IPI). It is clear now that IPI has no future: it has been usurped by its little sibling SCSI as the dominant high-performance disk interface. We discuss it here for historical reasons, as well as for those who have a substantial installed base of IPI devices. IPI, in addition to Fibre Channel, falls under the guidance of the T11 Technical Committee (http://www.t11.org).

The IPI standard was designed to facilitate optimization as much as possible, resulting in the highest possible performance; to that end, global optimization was built into the protocol. The fully implemented IPI model consists of four levels:

  • Level 0 defines mechanical and electrical considerations such as cables and connectors.

  • Level 1 defines primitive transfer details such as bus protocols.

  • Level 2 defines disk string protocols, which describe device-specific commands, timing, and volume addressing.

  • Level 3 defines I/O channel protocols, which include logical commands, command-queuing rules, buffering systems, etc.

A fully implemented IPI system consists of a number of IPI-3 host adapters (also called facilities), which connect to the host system. A host adapter is a complete I/O computer that is responsible for the management of a number of disk strings. Each string consists of one to eight disk drives, which are interfaced to the host adapter by an IPI-2 string controller. A disk can be connected to multiple strings, a condition known as multiporting. Up to sixteen string controllers interact with the host adapter over a high-performance IPI-3 bus. Like the host adapter, the string controller is vested with substantial intelligence; it is responsible for optimizing accesses along its string. Both the IPI-3 bus and IPI-2 string can extend up to fifty meters.

All of the intelligence in IPI is innate to the host adapter and the string controller; even primitive operations like bad-block mapping are done by the string controller. The disk drives are completely without intelligence and usually lack even local caches. As a result, an IPI disk must have a variety of conditions satisfied before it can perform a transfer:

  • The requesting string must not be otherwise occupied.

  • The string controller must have a free buffer to store the data.

  • The read/write head must be correctly positioned.

  • The data must be about to fly under the head.

Because the bus is used for essentially any transfer and is a resource that is shared between every device on the string, bus contention in IPI is a serious problem. For this reason, no more than three disks should reside on a single IPI-2 string; if the full eight disks are configured, severe performance problems arise.

In order to give the string controller enough information to optimize the request queue, IPI disks provide their rotational position and the radial position of the read/write heads. Each string controller then sorts a series of requests into an optimized order. Sometimes the string controller may be faced with a situation where it needs to retrieve data from two disks that are about to become ready to transfer. In such a case, the IPI-3 channel may be able to transfer the request to another string if the disk is connected to multiple strings, and if the operating system supports such a thing.[11] Incidentally, most IPI implementations do not fulfill the entire specification. For example, Sun did not fully implement the IPI-3 facility, instead treating each string controller as paired to exactly one host adapter. IPI performs best under massive load, when long queues form for each I/O device: if the queues are short or nonexistent, the intelligence in the IPI chain is wasted.

[11] Solaris does not.

5.2.3 SCSI

The Small Computer System Interface (SCSI, pronounced "scuzzy")[12] specification has been the mid-range workstation disk standard of the last fifteen years, and is clearly the dominant I/O interconnect in the modern workstation market. It is presently controlled by the T10 Technical Committee; see http://www.t10.org.

[12] This is something of a common joke amongst system administrators who have been fighting with a SCSI chain that is not operating properly. ("That scuzzy interface is screwed up again.")

The SCSI standard defines a peripheral bus model, which connects all the physical devices and is completely distinct from any other host bus, except for one intersection point. This intersection point is the host adapter, often called the SCSI controller. I refer to it as the host adapter, however, for reasons that will become obvious shortly.

Every device on a SCSI bus has a unique address that consists of two parts: a target identifier and a logical unit number (LUN). One analogy for this is mailboxes. Most mailboxes have a single identifying number (the target identifier) but some mailboxes have subunits, which are distinguished by the LUN (for example, a common set of mailboxes for an apartment building would have a single identifier but possibly many discrete mailboxes included within). For instance, the address "123 Anywhere Street, Apartment 918" would be equivalent to a target identifer of "123 Anywhere Street" and a LUN of "918." A request's target is specified by asserting a signal on a single line; as a consequence, an 8-bit SCSI bus has eight possible target identifiers.

Since it originally targeted small systems, the SCSI protocol was designed with an eye towards providing optimum performance at a low cost. In contrast to designs such as IPI, where much of the intelligence is vested in the controller, SCSI places the intelligence in the peripheral device itself. This greatly simplifies peripheral communication. Generally, this is handled by a microprocessor located in each device, called an embedded controller. Since this scheme means that the peripheral device controls itself rather than having an external controller, it doesn't make much sense to call a host adapter a SCSI controller.

One of the places that peripheral intelligence has greatly simplified operational procedures is defect management. Disk drives are very complex electromechanical devices, and bad blocks are inevitable. With host-intelligent systems such as IPI, the host was required to recognize disk errors and manage the defect list for every physical disk. With SCSI, this monitoring is accomplished at the disk level. In addition, the disk can monitor certain internal detail -- for example, when the defect management code is about to run out of free blocks -- and signal an impending failure to the host. One side effect, however, of peripheral-intelligent designs is that the intimate knowledge about a device resides with that device, rather than with a host controller; as a result, optimizations must be completed within the embedded controller, which can be complicated. From the host adapter's point of view, the resulting degree of abstraction allows a fantastically wide array of devices to interact over SCSI, but complicates optimization matters in large configurations.

If two devices wish to speak on the bus at the same time, arbitration is required. SCSI uses a simple scheme in which the highest-numbered target that requests the bus is given preference. For this reason, host adapters are almost always target 7.[13] It has also been common practice to assign slow devices such as tapes and CD-ROM drives high target numbers, so that they are not starved for bandwidth by fast devices. This is not generally a concern in the real world, however.

[13] This is even true (for backwards-compatibility reasons) in Wide SCSI implementations, which have 16 target identifiers.

5.2.3.1 Multi-initiator SCSI

Although it is the standard, it is not necessary to have the host adapter configured at target 7. This leads to an interesting case called multi-initiator SCSI, where a SCSI bus is shared between several systems. This can be used to share a pool of disks between two systems, which is useful in high-availability implementations. Unfortunately, it has some drawbacks:

  • There is no provision within the SCSI standard for handling pending operations when a bus reset occurs (e.g., if one initiator decides to issue a bus reset, a system panic occurs, or a transient electrical fault). This means that the state of the disks becomes undefined and data could be corrupted.

  • Although the standard defines a way for host adapters to "own" devices (via the reserve and release SCSI commands), there is no standardized way to break a reservation.

It is also possible to use multi-initiator SCSI to enable fast system-to-system communication over short distances, which is much safer than the high-availability disk sharing that multi-initiator schemes are usually used for. This functionality is available in Linux as a kernel module.

5.2.3.2 Bus transactions

Operations on a SCSI bus are quite abstract. They are accomplished by means of several operations, all of which consume bandwidth. When a target[14] wants to initiate a request, it must arbitrate the bus. Once control is granted, the initiator selects the target, and then specifies the command. The target now has to decide if it can service the request immediately. If it can (for example, the request is for a read that can be satisfied from the disk's local cache), the data is transmitted back with a message-in or message-out. If the request can't be immediately serviced (for example, the request is for a read that must actually be read from the platters), the target disconnects. At some later point, the target arbitrates for the bus, reconnects to the old initiator, and transmits the data (if appropriate) via message-in or message-out. The target then transmits the completion with a status message, and then disconnects. This results in six to eight state changes on the bus, which is busy except for the time between the disconnect and reconnect. Generally, the nondata parts of the transaction take about 140 microseconds. By comparison, transferring 2 KB of data takes about 100 microseconds. The overhead is fairly high because only the data portion is transmitted at the full burst speed: commands and status are always transferred in the slower asynchronous mode.

[14] This is generally the host adapter or a device wishing to communicate with the host adapter, but this is not always true; some systems (mostly those intended for real-time use) allow intradevice operations.

5.2.3.3 Synchronous versus asynchronous transfers

The SCSI standard defines two means of transferring data: synchronous and asynchronous. This refers to the way that the host adapter and its targets coordinate data transfer. In asynchronous transfers, data can be transmitted at an arbitrary time; the receiver and the sender need not be clock-synchronized. This kind of transfer necessitates that each byte be acknowledged, which imposes a practical limit on transmission speed. In synchronous mode, transfers may only start at specific clock phases, but this alleviates the acknowledgement problem that asynchronous transfers suffer from. The host adapter tries to negotiate the best way to communicate with devices when the targets are set up. If for some reason a synchronous speed cannot be negotiated with a target, the host adapter falls back to asynchronous transfers. You can find the negotiated synchronous transfer speed in Solaris via prtconf -v and by looking for the SCSI device driver. The next example is from a Fast SCSI-2 drive connected to an Ultra 1 workstation:

% prtconf -v

...

            esp, instance #0

               Driver properties:

                   name <target0-sync-speed> length <4>

                       value <0x00002710>.

...

In this case, the negotiated synchronous transfer speed to the device was configured as target 0; attached to this controller, it is 0x00002710 KB/second. This number is reported in hexadecimal; representing it in decimal and dividing by a thousand gives us the negotiated speed, which is 10 MB/second.

This is often worth checking. Using cables that aren't of sufficiently high quality or have improper termination issues can often cause devices to fall back to a slower speed. If you aren't getting the performance out of your SCSI chain that you think you should be, take a look here first.

5.2.3.4 Termination

SCSI operates electrically via a straightforward bus mechanism. Traffic from one device is sent out on the bus; the receiving device is responsible for pulling traffic destined for it off the bus. However, data that goes past the last device on the SCSI chain can be "echoed" back down the bus. This reflection phenomenon causes severe problems that must be addressed by termination. In a nutshell, termination is the addition of a set of resistors to a device in order to prevent signal reflection by "absorbing" data that would otherwise echo.[15] The rules for termination are, in theory, quite simple: terminate the first and last devices on a SCSI bus. Many modern disks will automatically terminate themselves as necessary. Termination problems are generally characterized by extremely slow data rates or devices that don't appear reliably (if at all). Note that, because of their different electrical designs, differential and single-ended terminators are not compatible (to learn more about the difference, see Section 5.2.3.6 later in this chapter).

[15] Termination is a fairly complex topic, and a full discussion is beyond the scope of this text.

5.2.3.5 Command queuing

When the original SCSI specification was drafted, performance was a secondary concern: the market at which the standard was being aimed was more interested in low cost than high performance. By the time that the SCSI-2 revision was issued, performance had become a far more important issue. In order to handle the problem of request optimization (that is, in which order to service requests), something needed to be done.

However, since each SCSI device has a high degree of autonomy, the host adapter lacks sufficient information to make real optimization decisions. Performing global optimization isn't possible, as was customary (and implemented in other standards, such as IPI). The SCSI-1 standard made this worse by permitting only one command to be pending to any given target; if only one request can be outstanding, no optimization can be performed!

This problem was addressed in SCSI-2 by adding command queuing, in which the target is able to accept multiple requests. It then processes them in the most advantageous order. Each request is assigned a tag (hence the synonymous term tag queuing), which serves as an identifier so that each entity can associate a given communication with a request. This process improved performance by allowing the overlap of bus transactions with physical activity, which saves a substantial amount of time, especially in applications that involve a lot of small disk writes. Unfortunately, command queuing is most effective when the disk is very heavily loaded, and generally you will get more out of spreading out the disk usage to reduce utilization than you will from the command queuing optimization. It has a limited impact on throughput for single-threaded tasks common to workstations.

5.2.3.6 Differential signaling

Let's say that you and a friend are both standing on a strip of beach somewhere, and your friend wants to send you a message. He has a lantern, and he holds it near his feet if he wants to send a zero and near his head if he wants to send a one. This works well as long as you can clearly distinguish which is the "high" position and which is the "low" one. If he gets in a boat and sails a mile offshore, you will have a much harder time determining which lantern position is high and which is low. However, if he takes two lanterns with him, and holds them at the same height for a zero, and raises one and lowers the other for a one, you will have a much easier time. The first mode of signaling, with only one lantern, is called single-ended; the second is called differential.

Low-voltage differential (LVD) signaling is a new electrical standard, introduced with the SCSI-3 specification, that accomplishes essentially the same thing as differential signaling but at a lower cost.

Differential signaling arose to alleviate single-ended SCSI's limited transfer range: single-ended, 20 MB/sec implementations are limited to just a meter and a half. Differential implementations can send data as far as 25 meters. Since these distances refer to the entire signal path -- including the cables between units, most of the traces on the embedded controller, and about an extra foot for every connector[16] -- a given length of cable will not always go as far as you might think.

[16] This is due to impedance issues across the connectors.

Differential signaling does not provide any performance improvement in and of itself. If the only difference between two SCSI buses is that one uses single-ended and the other uses differential signaling, they will perform exactly the same.

5.2.3.7 Bus utilization

Traditional performance recommendations suggest that no disk string should ever have a utilization of more than about 40%. This is based on old mainframe disk structures, which generally lacked caching at the disk; therefore, for a transaction to complete, the read/write head must have access to the controller in order to perform any transfer at all. If the bus is busy when the data flies under the head, the system must wait for the next window to perform the I/O -- at least 8.3 milliseconds later on a 7200 rpm disk. This is known as a rotational position sense miss (RPS miss). Because SCSI targets always have local caches, however, much higher bus utilization can be obtained before performance starts to suffer. Remember that SCSI is a high overhead architecture; each request can take six to eight state changes. For small I/O-size operations, this overhead starts to dominate performance (approaching 60% of total time). The crucial issue then becomes how the high SCSI overhead shapes performance.

For sequential operation, seek and rotation times are not significant in determining performance -- bus utilization problems are the core issue. For 2 KB I/O operations on a 20 MB/second chain, performance plateaus at about 8 MB/second, no matter how many disks are configured. The bus is capable of far greater performance, and using 64 KB I/Os results in a 95% reduction in the number of commands issued and maximum performance of about 17 MB/second. Here is a general guideline for configuring SCSI disk subsystems for sequential access:

  • Configure 20 MB/second chains so that no more than 4-8 drives are active at any one time. Large (~64 KB) I/O sizes will saturate the bus with 4 drives; small (~2 KB) I/O sizes will take 8 drives. (Remember that you will see much higher throughput with 64 KB I/Os than with 2 KB I/Os: the amount of data the bus can transfer is the sum of both actual data and command overhead, and with 2 KB I/Os the majority of data bits on the bus are commands!)

A disk array consisting of a number of disk drives is equivalent to the same number of disk drives directly connected to the host adapter.

Expect sequential throughput to be substantially less than the rated bus speed, particularly for small I/O sizes.

In random-access operation, a completely different phenomenon is observed. If accesses are truly random, the disks must seek before every I/O operation. Under these circumstances, the throughput of the disks drops dramatically (to something on the order of 250 KB/second). As would be expected, then, you can configure as many disks as you want: throughput scales linearly. Of course, at 250 KB/second per disk and fifteeen disk drives, you are only sustaining about 3.7 MB/second, so it would be surprising if you saw any bus throughput limitations.

  • If data access is truly random, bus performance is not a limiting factor at all. You can safely configure ~80 devices on a single 20 MB/second bus.

  • For real-world random access conditions, configure 1 or 2 disk arrays (16-64 devices) per 20 MB/second bus.

5.2.3.8 Mixing different speed SCSI devices

One common misunderstanding about SCSI is how faster devices are affected by slower devices on the same bus. This question usually comes up when someone installs a new, fast SCSI disk on the same chain as an older, slower disk. The SCSI standard is carefully designed so that the management of an individual device is isolated from its environment as much as possible. The host adapter negotiates an appropriate burst transfer speed and mode with each device on the bus individually, and the bus is driven to the limits of each device. However, some complexity does arise; for example, the utilization of the bus varies with the speed of the transfer, particularly if asynchronous transfers are involved. Slower devices tend to hog the bus, since they cannot select fast enough burst transfer speeds to get off the bus expediently. This is almost never an issue in a workstation environment, but when many devices are present on a system, it is often a good idea to segregate low speed devices onto a separate bus to maintain throughput elsewhere.

5.2.3.9 SCSI implementations

In the spring of 1979, Shugart Associates began defining the Shugart Associates System Interface (SASI), which was subsequently released to the public domain in order to encourage disk controller companies to use the protocol. In 1980, SASI was placed before the ANSI Committee X3T9.3, in order to have it accepted in place of IPI (see Section 5.2.2 earlier in this chapter).

In the summer of 1981, NCR began exploring the possibility of using SASI instead of a proprietary interface that they had recently defined. This idea fell through due to fears that technical improvements would induce market confusion. In the early fall of 1981, however, Optimem (a Shugart subsidiary) requested the same technical improvements that NCR had. In a move to accept the inevitable, Shugart reapproached NCR, and a series of meetings were held to more fully document SASI and add several features drawn from NCR's proprietary interface. In December of 1981, NCR and Shugart jointly requested that X3T9.3 grant time at their next meeting in February of 1982 to consider the joint proposal. X3T9.3 responded by breaking the meeting into two parallel sessions, one for IPI and one for SASI. At the conclusion of that meeting, the sessions rejoined and decided that SASI should be assigned to an existing task group, X3T9.2, which had no current project. In April of 1982, X3T9.2 met and drafted a formal project proposal for the Small Computer System Interface (SCSI), which was to be based on SASI. Throughout 1982 and 1984, X3T9.2 met regularly, fully documenting SCSI with contributions from NCR (which proposed command sets for tapes, processors, and printers) and Optimem (which proposed an optical write-once, read-many, or WORM, command set). In April of 1983, NCR announced the NCR-5385, the first SCSI protocol chip; it was followed in April 1984 by the NCR-5380, which added significant hardware support for inexpensive SCSI implementations.[17] Also in April of 1984, X3T9.2 voted to forward SCSI to its parent committee in order to begin the lengthy ANSI standards approval process. Approval was gained in June of 1986, and SCSI was formally ANSI X3.131-1986.

[17] The NCR-5380 was the chip initially used by Apple Computer for the SCSI bus on the Macintosh Plus.

In July of 1985, a special working group began a series of meetings to define enhancements to the SCSI command set for disks, later known as the Common Command Set (CCS). In June of 1986, X3T9.2 began the SCSI-2 project, which endeavoured to add support for the CCS and other enhancements into the SCSI protocol. 1986 to 1989 was a period of intense work, and the SCSI standard grew from 200 pages to nearly 600 pages. SCSI-2 was forwarded for ANSI approval in February of 1989, and approved in September of 1990.

Further development work in SCSI, specifically SCSI-3, involved splitting the effort into multiple task groups, each focusing on specific parts of the protocol.

Although SCSI-2 is completely backwards-compatible with SCSI-1, SCSI-3 is not completely compatible with SCSI-2 due to some aggressive hardware-oriented extensions. However, software is not affected.

The SCSI-3 standard has been adopted piecemeal; many vendors have chosen to implement some of the more compatible parts already. For an overview of the SCSI implementations, please consult Table 5-4.

Table 5-4. SCSI implementation overview

SCSI-1

SCSI-2

SCSI-3

Media

Copper

Copper

Copper

fibre

Topologies

Tree

Tree

Multi-initiator tree

Tree

Multi-initiator tree

Arbitratrated loop

Fabric

Signalling

Single-ended

Single-ended

Differential

Single-ended

Differential

LV differential

Command queuing

None

Optional

Optional

Transfer modes

Synchronous

Asynchronous

Synchronous

Asynchronous

Synchronous

Asynchronous

Transfer clocking

5 MHz

5 MHz

10 MHz

5 MHz

10 MHz

20 MHz

Transfer width

8-bit

8-bit

16-bit

8-bit

16-bit

I find myself answering a lot of questions about exactly how fast a given standard is. A lot of people I know have ended up building tables that look remarkably like Table 5-5 and Table 5-6. I hope you find these useful.

Table 5-5. Deciphering SCSI variants

Standard

Clock (MHz)

Width (bits)

Max. rate (MB/sec)

Max. devices

Max. length[18]

SCSI-1

SCSI-1

5

8

5

8

6.0

Fast

SCSI-2

10

8

10

8

3.0

Wide

SCSI-2

5

16

10

16

3.0

Fast/Wide

SCSI-2

10

16

10

16

3.0

Ultra (fast-20)

SCSI-3

20

8

20

8

SE: 1.5 Diff: 12.5

Ultra Wide (fast-40)

SCSI-3

20

16

40

16

SE: 1.5 Diff: 12.5

Ultra2

SCSI-3

40

8

40

16

SE: 1.5 Diff: 12.5

Ultra2 Wide (fast-80)

SCSI-3

40

16

80

16

SE: 1.5 Diff: 12.5

Ultra160

SCSI-3

80

16

160

16

Diff: 12.5

[18] In meters, for synchronous operation. Unless specified, for single-ended (SE) configurations only.

Table 5-6. Deciphering SCSI connectors

Connector

Pins

Form factor

Used for

SCSI-1

50

D-shell connector 3 rows of pins

SCSI-1

DB-25

25

D-shell connector 2 rows of pins

SCSI-1, narrow SCSI-2

Primarily used on Macintosh

Centronix 50

50

Centronix connector

SCSI-1, narrow SCSI-2

Micro 50

50

Micro-type connector 2 rows of pins

Narrow SCSI-2, narrow SCSI-3

Micro 68

68

Micro-type connector 2 rows of pins

Wide SCSI-2, wide SCSI-3

SCA

80

Micro-type connector 2 rows of pins

SCSI-2, SCSI-3

Internal attachment only

"Universal" SCSI connector: integrated power and ID select

VHDC

68

Very High Density connector

Differential SCSI-3

HD-60

60

Micro-type connector with a bar instead of pins

Narrow SCSI-2

Old IBM SCSI adapters only

HD-68

68

Micro-type connector with a bar instead of pins

Wide SCSI-2

Old IBM SCSI adapters only

5.2.4 Fibre Channel

The Fibre Channel (FC) specification, defined by ANSI X3T9.3, uses SCSI-like commands, typically over fiberoptic cable rather than copper wires. Fibre Channel standards are the purview of the T11 Technical Committee (http://www.t11.org).

Fibre Channel is a full-duplex medium, meaning that it can support bidirectional data flow; it requires a fiber strand for each direction. Although it is defined by its own standard, it has been brought under the SCSI-3 umbrella. Fibre Channel has three very different topologies:

Point-to-point

One device connects to exactly one other device.

Arbitrated loop

Often called FC-AL, the devices and one or more hosts are connected in a ring using many point-to-point links.

Fabric

Switches and hubs are used to create a complex network in which there may be multiple paths from a given host to a given device. This is becoming known as storage-area networking, and is probably the direction enterprise-class storage will move in the future. Unfortunately, a discussion of the performance of storage area networks (SANS) is far too complicated for this text.

Fibre Channel devices are assigned a unique address, called the world wide name (WWN). Since devices can be connected into arbitrary fabrics, the common practice is to assign each device a completely unique WWN, much like Ethernet addresses are assigned. The Fibre Channel standard defines three classes of signaling, which are not interchangeable. They are usually described in terms of their data speed: 25 MB/second (FC-25), 50 MB/second (FC-50), or 100 MB/second (FC-100). The storage industry as a whole initially deferred acceptance of Fibre Channel until the arrival of FC-100 parts, but it is now in widespread use. In general, Fibre Channel disks should work just like SCSI disks. Be sure to follow any configuration rules provided by the vendor carefully.

Fibre Channel host adapters typically have two optical link modules, which enable the host adapter to connect to two independent Fibre Channel buses; the connection is managed by the serial-optical coupler, or SOC (typically called a GBIC). A typical SOC can handle up to 255 pending requests, which can occasionally become a limitation for single-connection interfaces and is usually a bottleneck for biported interfaces. In order to work around this restriction, it is vital that FC-interfaced devices be driven at large I/O sizes (8-64 KB). In light of this fact, the second port is probably most useful for attaching archival or other low-use devices.

One of Fibre Channel's most useful capabilities is the removal of distance restrictions imposed by the limitations of copper wires; the FC standard allows cable runs of up to ten kilometers. This allows the geographic dispersal of storage, which has remarkable implications for disaster recovery.

5.2.5 IEEE 1394 (FireWire)

Just like Fibre Channel, the IEEE 1394 specification has also been brought into the SCSI-3 fold. IEEE 1394, also known as FireWire, was originally developed by Apple Computer. It is rapidly entering widespread use in the midrange market for desktop systems.

By far the most important thing to consider in evaluating IEEE 1394 peripherals is that the bus speed is quoted in megabits per second -- not megabytes per second -- just like network interconnects. Currently, it is available in 100, 200, and 400 Mbits per second variants. IEEE 1394 supports the interconnection of up to 63 devices to ausuallysssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss


ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss-ssssssssssssssssssssssssssssssssssss:sssssssssssscistsssssssssi

Foxt">MicrsssssssCell">

Olme reasoesssssssTsssssssssssssssCeNossssssssssssssssssssssssssssTnssss clas width=b the dsssssssssssssSCSI-2sssssssD classsssssstransferrinSPACING="0sss sssssssssssssP clasiTableP class="do>5.2.sssssssssrformance haheight f.onfigure ~80 devices on a single 20 FIG-sssFrk aro sssss 139wssssssssssssssssdocTssssscistsssssssssi The SCS0 piecemea482es EIGHTea58sssSRCsssssssss05960e usX/orkssspf2_05e ssssssALTeaorkssspf2_05e ssssssssCENTERdersssssssssssssssssssssssssssss4T-5-310291">10 messss200, anI varssr">Arbi10Point-to-point 3/span>sssssssssssssssssssssssssssssssssssssled the ylass=sssssupssssssP class=wDNLCDiffessssssssssplcome a liate th physup classssss dnctlysssssssssssssssssssssssssssssswDNLCsssg025sssssssssssssssn ncts"E="ssssssssi ass=< claassgws are connected in a ring ssss/pnnerdmany ) wDNLCsintonnectisableCell"> FCss=="docTssiby loosssssssssssss"> bly the ">MicsCell">

Centr cssssssstems. ssssssssd the ylsssssymbol sssssssas by dbusddiniby aren't D30ssssstossslsssl">

sss0ssssssssssssessss8,000ss=wsossssss0the "hi0 0lass=cTablsssssssssaP c ssssendecText">8 0lx 17)00nd port is probably most useful for 2/O sizes (8-64 KB). In2ation than you will2from the.)e[18] In meters, for synchronouoneWire)P 139wss; twDNLCsintonnects disk'sbitrary ts/TD>

Fofull burpan classsst">SCSI-leHeadedsssssssssssssssssssms ass=i/lssTT>nC sld-le sssss hosts are connected in a ring slo -amany pText">dsssssssssssssssssssmsass=i/lmany sssdbkanssss6:44:55 111 s are connected in aBold">603many sssss203ssssssor each direction. Although it is dewosssssssssssssI>world nctsstconf nctlyssssssssssjust lisk's loc>Switches P classthe FC s clfor

SCSI-1, narr32 SCSI-2

<<Point-to-point 34span>sssssUD>Switchsssi Point-to-point 35eWire)Tabrapisssssssssssssata speed: 25 MB/second (FC-2tunately,or">Point-to-point 36eWire) sssssout 100 missssss classnnection oforldssssssadapter e probleer">

Forworld nctsssssssssA NAME="itchsled the ysr sdisstconfssssssadapter ltelhonsssssssssssss class="ssc> IEEE 139ssss">Fitor 2 rodssssssssbdar 2