projects techreports press lab location staff
citi top.2 top.3
citi mid.3
bot.1 bot.2 bot.3

pNFS/PVFS2 Documentation

Linux pNFS/PVFS2 Installation Instructions

Step 1: pNFS Kernel
  • Ensure you have access to a built pNFS kernel source tree.
Step 2: Compile, Boot, and Verify Kernel and NFSv4.
  • Compile and install kernel on every client, data server, and MDS. Instructions are located here.
  • Ensure NFSv4 is working properly. For example, export and mount a local directory through the loopback device.
Step 3: Ensure you have a working PVFS2 file system.

Step 4: Export the pvfs2 file system via NFSv4 on the MDS.

Step 5: Install pvfs2 layout driver.

  • Compile and install PVFS2 server code. Execute the following instructions in the pvfs2 layout driver directory:

    ./configure -with-kernel=/cvs/pnfs -prefix=/usr/local/bin/pvfs2-layout --disable-server --disable-karma
    /cvs/pnfs is the compiled pNFS kernel source tree.
    /usr/local/bin/pvfs2-server is the installation target.
    make install
    make KMOD_DIR=/usr/local/bin/pvfs2-layout kmod_install

  • More PVFS2 specific instructions can be found here.
Step 6: Create /etc/pvfs2tab file on every client.

File contains

tcp://(mds serrver name):3334/pvfs2-fs
Step 7: Mount PVFS2 file system using pNFS
  • Execute the following commands on the client:

    insmod /usr/local/bin/pvfs2-layout/pvfs2-pnfs.ko
    /usr/local/bin/pvfs2-layout/sbin/pvfs2-client -p /usr/local/bin/pvfs2-layout/sbin/pvfs2-client-core
    mount -t nfs4 (mds server name):/ /mnt/nfs4/

Step 7: Stopping pvfs2 layout driver.
  • Execute the following commands on the client:
  • umount /mnt/pnfs
    killall /usr/local/bin/pvfs2-layout/sbin/pvfs2-client
    rmmod pvfs2_pnfs

Layout Driver Policies

Several /proc/sys/pvfs2-pnfs/ variables exist to control the behavior of the PVFS2 layout driver. They are:
  • blocksize - Size (in bytes) of reads and writes to storage nodes. Currently the maximum size is 32768.
  • write/read_threshold - Threshold (in bytes) under which writes/reads will be sent to the NFSv4 server. Value is -1 to always use direct I/O through the layout driver.
  • use_pagecache - Value of 0 to avoid using page cache, all writes/reads go straight to PVFS2 storage nodes (this is the default behavior of unmodified PVFS2). Value of 1 to use page cache. This enables PVFS2 with a data cache, readahead, and a writeback cache.

Debugging Help

  • NFS debugging:

    echo 32767 > /proc/sys/sunrpc/nfsd_debug
    echo 32767 > /proc/sys/sunrpc/nfs_debug

  • pNFS layout driver debugging:

    // Before layout driver is loaded
    export PNFS_DEBUGMASK=all
    // Uses log file /tmp/pnfs-client.log

  • PVFS2 debugging, see PVFS2 website

Known Problems

  • Sometimes interrupting an application while writing/reading will cause the application to hang.

Please help improve these instructions, email projects | techreports | press | lab | location | staff Email address
or call +1 734 763 2929
Copyright © 1996-2013
The Regents of the University of Michigan