projects techreports press lab location staff
citi top.2 top.3
citi mid.3
bot.1 bot.2 bot.3

pNFS/PVFS2 Documentation

Linux pNFS/PVFS2 Installation Instructions

Note: If you get confused at any point, verify your setup with the diagram below.

Step 1: Download and Untar

  • Download and untar pNFS kernel, PVFS2 Server, and PVFS2 Layout Driver.
Step 2: Compile, Boot, and Verify Kernel and NFSv4
  • Compile and install pNFS kernel on every client, data server, and MDS. Instructions are located here under Instructions.
  • Ensure NFSv4 is working properly. For example, export and mount a local directory through the loopback device.
Step 3: Compile and Verify PVFS2 storage system
  • Compile and install the PVFS2 server code. Follow the instructions on the PVFS2 website.
  • Example:

    ./configure -with-kernel=/path/to/kernel -prefix=/usr/local/bin/pvfs2-server
    make install
    make KMOD_DIR=/usr/local/bin/pvfs2-server kmod_install

  • To verify, create a pvfs2 file system and access it. All instructions are on the PVFS2 website
  • Example (easiest to run all commands as root):

    // Define PVFS2 file system and create config files
    // Run on MDS
    /usr/local/bin/pvfs2-server/bin/pvfs2-genconfig /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf
    // Copy /etc/pvfs2 directory to every data server
    // Create /etc/pvfs2tab file with contents:
    tcp://"mds_server_hostname":3334/pvfs2-fs /mnt/pvfs2 pvfs2 default 0 0
    // Create storage space
    // Run on each data server and MDS
    /usr/local/bin/pvfs2-server/sbin/pvfs2-server /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf-hostname -f
    // Start server
    // Run on each data server and MDS
    /usr/local/bin/pvfs2-server/sbin/pvfs2-server /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf-hostname
    // Test filesystem
    // Run on MDS and ensure positive result
    /usr/local/bin/pvfs2-server/bin/pvfs2-ping -m /mnt/pvfs2
    // Load kernel interface
    // Run on MDS
    insmod /usr/local/bin/pvfs2-server/pvfs2.ko
    /usr/local/bin/pvfs2-server/sbin/pvfs2-client -p /usr/lo:cal/bin/pvfs2-server/sbin/pvfs2-client-core
    mount -t pvfs2 tcp://"mds_server_hostname":3334/pvfs2-fs /mnt/pvfs2
    // Now run ls, cat, etc in /mnt/pvfs2 to ensure you have successfully
    // created a pvfs2 file system.

Step 4: Export PVFS2 storage system with NFSv4
  • Create /etc/exports file on MDS.
    /mnt/pvfs2  *(rw,sync,fsid=0,insecure,no_subtree_check)
  • Run commands on MDS:

    rpc.nfsd 8
    exportfs -r

Step 5: On clients, compile and install PVFS2 layout driver.
  • Example:

    ./configure -with-kernel=/cvs/pnfs -prefix=/usr/local/bin/pvfs2-layout --disable-server
    make install
    make KMOD_DIR=/usr/local/bin/pvfs2-layout kmod_install

Step 6: Verify you can mount exported PVFS2 file system via standard NFSv4
  • On client:

    mount -t nfs4 "mds_server_name":/ /mnt/nfs4/

  • If successful:

    umount /mnt/nfs4/

Step 7: Mount PVFS2 file system using pNFS
  • Copy /etc/pvfs2tab file to every client
  • Example:

    // On client
    insmod /usr/local/bin/pvfs2-layout/pvfs2-pnfs.ko
    /usr/local/bin/pvfs2-layout/sbin/pvfs2-client -p /usr/local/bin/pvfs2-layout/sbin/pvfs2-client-core
    mount -t nfs4 "mds_server_name":/ /mnt/nfs4/

System Configuration Diagram

Layout Driver Policies

Several /proc/sys/pvfs2-pnfs/ variables exist to control the behavior of the PVFS2 layout driver. They are:
  • blocksize - Size (in bytes) of reads and writes to storage nodes.
  • write/read_threshold - Threshold (in bytes) under which writes/reads will be sent to the NFSv4 server. Value is -1 to always use direct I/O through the layout driver.
  • use_pagecache - Value of 0 to avoid using page cache, all writes/reads go straight to PVFS2 storage nodes (this is the default behavior of unmodified PVFS2). Value of 1 to use page cache. This enables PVFS2 with a data cache, readahead, and a writeback cache.

Debugging Help

  • NFS debugging:

    echo 32767 > /proc/sys/sunrpc/nfsd_debug
    echo 32767 > /proc/sys/sunrpc/nfs_debug

  • pNFS layout driver debugging:

    // Before layout driver is loaded
    export PNFS_DEBUGMASK=all
    // Uses log file /tmp/pnfs-client.log

  • PVFS2 debugging, see PVFS2 website

Known Problems

  • Sometimes interrupting an application while writing/reading will cause the application to hang.

Please help improve these instructions, email projects | techreports | press | lab | location | staff Email address
or call +1 734 763 2929
Copyright © 1996-2013
The Regents of the University of Michigan