ASCI Cluster Information
Node Chart
| Fast Ethernet Nodename |
Fast Ethernet IP |
Gigabit Nodename |
Gigabit IP |
| iota1.citi.umich.edu |
141.211.133.200 |
iota1g.citi.umich.edu |
141.211.133.201 |
| iota2.citi.umich.edu |
141.211.133.202 |
iota2g.citi.umich.edu |
141.211.133.203 |
| iota3.citi.umich.edu |
141.211.133.204 |
iota3g.citi.umich.edu |
141.211.133.205 |
| iota4.citi.umich.edu |
141.211.133.206 |
iota4g.citi.umich.edu |
141.211.133.207 |
| iota5.citi.umich.edu |
141.211.133.208 |
iota5g.citi.umich.edu |
141.211.133.209 |
| iota6.citi.umich.edu |
141.211.133.210 |
iota6g.citi.umich.edu/Local |
141.211.133.211/Local |
| iota7.citi.umich.edu |
141.211.133.212 |
iota7g.citi.umich.edu/Local |
141.211.133.213/Local |
| iota8.citi.umich.edu |
141.211.133.214 |
iota8g.citi.umich.edu/Local |
141.211.133.215/Local |
| llnl1.citi.umich.edu |
141.211.133.119 |
Same |
Same |
| llnl2.citi.umich.edu |
141.211.133.120 |
Same |
Same |
Iota Hardware
- Dual 1.7 GHz Pentium 4 processors with a 256 KB cache
- 2 GB of RAM
- Seagate 80GB, 7200 RPM hard drive with an Ultra ATA/100 interface and a 2 MB cache (Nodes 5-8 have an additional 120GB drive)
- 3Com 3C996B-T gigabit card (Nodes 6-8 have 2 cards) (64Mbit/66Mhz)
- PCI-X(133Mhz) risers for the gigabit cards
- 2 64bit/66Mhz PCI slots
- Front Side Bus 400Mhz
LLNL Hardware
- 1.4 GHz Intel Xeon processors with a 256 KB cache
- 1GB RAM
- Adaptec 40GB, 10K RPM SCSI hard drive using Ultra 160
- AceNIC gigabit Ethernet card (64Mbit/66Mhz)
- Myrinet card running at 66Mhz
- Note: I'm not sure if the cards are 66Mhz, or if it is the bus, or if only 1 card is 66Mhz and the other is 133Mhz. The only way to find out is to open up the machine. Type 'lspci -vvv' for more info.
Software
- All iota and llnl machines are currently running redhat 9 with various kernels
Switch
- IntraCore 35160 gigabit switch
Power
- APC 16 Port Powerbar
- URL: http://iota-power.citi.umich.edu
- IP: 141.211.133.239
Console
- Hostname:ccc.citi.umich.edu
- IP: 141.211.133.14
- To connect type: ssh username:iota<#>@ccc
GPFS
- Currently, GPFS is installed over iSCSI to fas960-1 on iota4-8 on sda
- I used the following script to install the rpms
- It is currently set for testing, not performance as the block size is 16KB and the pagepool is only 32MB.
- I used the following script to create it
- You can use the following script script to delete it
Using the machines
Type 'mount /nfs' to retrieve plymouth home directories
Here are sample iota .config files:2.4 2.6
Here are sample llnl .config files:2.4 2.6
Use 'authconfig' to set level of security access (Kerberos,etc)
For test purposes, if you want to control the CPU frequency, you must enable 'CPU Frequency Scaling' in the kernel and then modify the value in /proc/sys/cpu/0/speed
'hdparm /dev/hdX' adjusts disk parameters. Need to ensure DMA, etc are enabled. 'hdparm -[t|T] /dev/hdaX' (noncached/cached) will perform tests.
The best driver for the 3Com cards is the Broadcom BCM5701 driver (not the Tg3 driver). It is available here
- This driver is currently installed as a module. A patch exists for the 2.6.0 kernel which should work with later kernels, but modules are easier with the rapid number of kernels used in testing.
- To manually install a module, you must first create it (.o for 2.4 and .ko for 2.6), put it in /lib/modules/'uname -r'/kernel/drivers/net/ and then add it to the /lib/modules/'uname -r'/modules.dep file. The eth1/eth2 module must then be modified in the /etc/modules.conf (for 2.4) or /etc/modprobe.conf (for 2.6)
Some very basic kernel profiling help. I recommend reading the README too.
Some useful performance monitoring tools are:
- 'top' -individual processor statistics and memory
- 'sar' -Interrupts(-I), network(-n DEV), per cpu (-U ALL or -P ALL)
- 'vmstat' -A wonder tool
- 'iostat' -x is useful for disk info
Setting up the serial port
ISCSI 3.4.1 is installed on iota4-8. Use /etc/init.d/iscsi stop/start. The config file /etc/iscsi.conf is here.
A sample /etc/sysconfig/network-scripts/ifcfg-eth1 file.
/usr/local/bin/iozone_ssh is installed on each machine. It is the same as iozone, but uses ssh instead of rsh for throughput tests
Here is a lilo.conf file for iota4-8
Here is a lilo.conf file for iota1-3
Here is a lilo.conf file for llnl1 machines
Here is a lilo.conf file for llnl2 machines
If you have comments or suggestions about our NFSv4 work with ASCI, email
asci @ citi.umich.edu
