# dmr's notes on doing a PMP install on top of RH FC1: Globus 2.2.4 GK/2.4-GARA, # GARA 1.2.2, krb5, OpenAFS client software, our NTAP software, etc. # # richterd@citi.umich.edu ################################################################################### 0. intro. . hello this document explains our PMP installation/prep procedure, but it also has my random notes on various experimental parts. . hardware requirements: . we assume a 802.1q-savvy NIC . software requirements: . glibc 2.3+ . krb5 client stuff [FAILED] ftp://216.254.0.38/linux/redhat/updates/9/en/os/SRPMS/krb5-1.2.7-14.src.rpm [WORKED] http://web.mit.edu/Kerberos/www/dist/krb5/1.3/krb5-1.3.2-i686-pc-linux-gnu.tar . misc: . la0:/etc/krb.conf . la0:/etc/krb5.conf . openafs stuff http://openafs.org/dl/openafs/1.2.11/fedora-1.0/openafs-1.2.11-fc1.0.1.i386.rpm http://openafs.org/dl/openafs/1.2.11/fedora-1.0/openafs-client-1.2.11-fc1.0.1.i386.rpm http://openafs.org/dl/openafs/1.2.11/fedora-1.0/openafs-compat-1.2.11-fc1.0.1.i386.rpm http://openafs.org/dl/openafs/1.2.11/fedora-1.0/openafs-devel-1.2.11-fc1.0.1.i386.rpm http://openafs.org/dl/openafs/1.2.11/fedora-1.0/openafs-kernel-1.2.11-fc1.0.1.i386.rpm http://openafs.org/dl/openafs/1.2.11/fedora-1.0/openafs-kpasswd-1.2.11-fc1.0.1.i386.rpm http://openafs.org/dl/openafs/1.2.11/fedora-1.0/openafs-krb5-1.2.11-fc1.0.1.i386.rpm http://openafs.org/dl/openafs/1.2.11/fedora-1.0/openafs-kernel-source-1.2.11-fc1.0.1.i386.rpm . misc: . la0:/usr/vice/etc/CellServDB . la0:/usr/vice/etc/ThisCell . kx509 (for kx509 and kxlist) http://www.citi.umich.edu/projects/kerb_pki/kx509.dist20031111.tar.gz . KeyNote 2.3 http://www.cis.upenn.edu/~keynote/ . NOTE: keynote also needs the source code for the installed openssl (0.9.7a, by default in FC1) http://www.openssl.org/source/openssl-0.9.7a.tar.gz . NOTE: keynote also needs rsaref.tar.Z. google for it and toss it in /usr/local/src/. . GPT (Grid packaging toolkit) 2.2.5 (for Globus 2.2.4) and 3.0.1 (for Globus 2.4) http://www-unix.globus.org/toolkit/release-archive.html http://www.globus.org/gt2.4/download.html . Globus 2.2.4, resource manager client and server source tar.gzs http://www-unix.globus.org/toolkit/release-archive.html . Globus 2.4, resource manager client and server source tar.gzs http://www.nsf-middleware.org/NMIR3/download.asp http://www.globus.org/gt2.4/download.html . aglo's custom GARA 1.2.2 [email richterd@citi.umich.edu for a current CVS snapshot] . iperf 1.7.0 http://dast.nlanr.net/Projects/Iperf/iperf-1.7.0-source.tar.gz . NANOG traceroute ftp://ftp.login.com/pub/software/traceroute/traceroute.c gcc -o traceroute -lm -lresolv traceroute.c 1. base install. . used Redhat's Fedora Core 1 and did a clean install on l99.citi.umich.edu . no firewall, grub bootloader . after the install, i enabled ntpd . set default runlevel to 3 . changed my UID to my umich one, for AFS . ports 22, 111, and 32770 are currently receptive . ssh version: 3.6.1p2 (too old; will move to 3.7.1p2 later) . ssl version: 0.9.7a (right on; will work with current kx509) . glibc version: 2.3.2 (ok) 2. kerberos. . downloaded from the first krb5 URI above, but rpmbuild --recompile failed when building telnet (ugh) because it uses old vararg.h instead of the new stdarg.h. ack. so, despite having fixed their Makefile to remove telnet, i'm not seeing enough reason to stay with 1.2.7-14 if i can just use MIT's tarball of binaries for 1.3.2. . downloaded the second URI from MIT . sudo tar xvzf krb5-1.3.2-i686-pc-linux-gnu.tar.gz -C / . note that everything seems to go under /usr/local/, instead of /usr/kerberos/ or whatever. . copied over krb{,5}.conf to /etc/ . kinit, kxlist work. 3. openafs. . downloaded all of the RPMs from above into /usr/local/src. . sudo rpm -ivh openafs-*.rpm . copied over CellServDB and ThisCell to /usr/vice/etc . sudo /etc/init.d/afs start . i can auth, see files, etc. 4. kx509. . downloaded the release from the above URI into /usr/local/src . cd kx509 . src/configure --with-krb5=/usr/local . make . copied kx509 and kxlist to /usr/local/bin 5. globus 2.4 (for gara). . NOTE: installation instructions are at http://www.globus.org/gt2.4/install.html . downloaded 2.4 src for client and server . downloaded GPT 3.0 to /usr/local/src . unpacked GPT . export GPT_LOCATION=/usr/local/src/gpt-3.0.1 . cd $GPT_LOCATION . ./build_gpt . okay, that's done . export GLOBUS_LOCATION=/usr/local/src/globus-2.4 . copied the globus src .tar.gzs into $GLOBUS_LOCATION . cd $GLOBUS_LOCATION . $GPT_LOCATION/sbin/gpt-build globus-resource-management-client-2.4.3-src_bundle.tar.gz gcc32dbgpthr . NOTE: this takes approximately forever. . $GPT_LOCATION/sbin/gpt-build globus-resource-management-server-2.4.3-src_bundle.tar.gz gcc32dbg . NOTE: this takes slightly longer than forever. . NOTE: `gcc32dbgpthr' isn't a valid option for the server, as it is with the client. . $GPT_LOCATION/sbin/gpt-postinstall . sudo /usr/local/globus-for-gara/setup/globus/setup-gsi . first, say "y" . next, say "q" . XXX at this point, one normally sets up /etc/grid-security/, which involves making signed certificates (by the KDC) for the PMP. that is, of course, if this installation is going to be your globus_gatekeeper installation. but we'll probably use globus 2.2.4 for that. [IGNORE; this is an old binary installation that bailed -- just keeping the notes] . /usr/local/src/gpt-3.0.1/sbin/gpt-install globus-resource-management-sdk-2.4.3-i686-pc-linux-gnu-bin.tar.gz . source /usr/local/globus-for-gara/etc/globus-user-env.sh . /usr/local/src/gpt-3.0.1/sbin/gpt-postinstall . /usr/local/src/gpt-3.0.1/sbin/gpt-build gcc32dbg -nosrc 6. keynote. . okay, keynote's a little weird. . i downloaded openssl-0.9.7a source and rsaref.tar.Z, in expectation of the next steps. . first, an overview: to build keynote, i need librsaref.so. i found some stuff in /usr/local/src/openssl-0.9.7a/demos/engines/rsaref . it wanted the old rsaref 2.0 (google: rsaref.tar.Z) library. follow the `README' file in the above dir. then, i add /usr/local/openssl-0.9.7a/demos/engines/rsaref to ld.so.conf so `librsaref.so' gets found for keynote. . now, the steps: . unpacked openssl-0.9.7a sources -- note that i'm not going to build them; it's already installed on FC1. . get rsaref.tar.Z . pushd /usr/local/openssl-0.9.7a/demos/engines/rsaref/ . tar xvZf /usr/local/src/rsaref.tar.Z -C /usr/local/openssl-0.9.7a/demos/engines/rsaref/ . make gnu . rm librsaref.so.gnu (empty file that ldconfig doesn't like) . sudo vi /etc/ld.so.conf (add path so librsaref.so gets found by gara) . sudo ldconfig . okay, now we've built librsaref.so. let's get on to keynote. . cd /usr/local/src/keynote-2.3 . ./configure . had to edit Makefile with different paths for: AR, TR, SSLEAY . had to edit Keynote's Makefile after configure to change: from LIBS = -L. -lkeynote -L/usr/local/ssl/lib -L/usr/lib -L/usr/local/lib -lm -lrsaref -lcrypto to LIBS = -L. -lkeynote -L/usr/local/ssl/lib -L/usr/lib -L/usr/local/lib \ -L/usr/local/openssl/demos/engines/rsaref -lm -lrsaref -lcrypto . cheaped out and did `sudo ln -s /usr/include/openssl /usr/include/ssl` to get to the headers . ./configure --prefix=/usr/local/src/keynote-2.3 . make crypto . make test . make test-sig 7. gara. . acquire CITI's GARA sources (email us) . cd /usr/local/src . cvs co ntap2/gara-1.2.2 . mv ntap2/gara-1.2.2 . && rm -rf ntap2/CVS && rmdir ntap2 . cd gara-1.2.2 . export GLOBUS_LOCATION= // <-- whatever you set it to above . export GPT_LOCATION= // <-- whatever you set it to above . export CFLAGS="-g -DGENERIC_RSL -DCLIENT_INFO -DCHECK_AUTHORIZATION -DDEBUG -DDEBUG_MOD_PTS -DPERMIS -DDEBUG_PERMIS ${CFLAGS}" . export GLOBUS_CFLAGS="-I/usr/local/src/globus-2.4/include/gcc32dbgpthr/openssl -I/usr/local/src/globus-2.4/include/gcc32dbgpthr" . ./configure --prefix=/usr/local/src/gara-1.2.2 --with-globus-prefix=/usr/local/src/globus-2.4 \ --with-globus-flavor=gcc32dbgpthr --enable-lram_diffserv --enable-rm_diffserv \ --enable-gara_diffserv --enable-rm_network --enable-slot_manager . make . NOTE: the first `make` might die really fast. try it once more. . yikes. turns out that the gara build is just falling all over itself. . much failure brings to light a libtool version-incompatibility issue: FC1 ships with a libtool that gara's build stuff doesn't invoke quite right. . we need: [richterd@la0 libraries]$ libtool --version ltmain.sh (GNU libtool) 1.4.2 (1.922.2.54 2001/09/11 03:33:37) . that's the version that shipped with redhat 7.3, anyway, and gara likes it. in the future, we could probably tailor gara's makefiles to use the the older libtool, while leaving the newer version installed for everything else to use. for now, i put libtool in by hand and just moved the newer version aside. 8. globus-2.2.4 (server stuff, mainly). . okay, now we need to build Globus 2.2.4 (for the gatekeeper & friends) . mkdir -p /usr/local/src/globus-2.2.4 . cd /usr/local/src/globus-2.2.4 . mv ../globus-resource-management-*.tar.gz . . export GLOBUS_LOCATION=/usr/local/src/globus-2.2.4 . export GPT_LOCATION=/usr/local/src/gpt-2.2.5 . cd $GPT_LOCATION . ./build_gpt . cd $GLOBUS_LOCATION . $GPT_LOCATION/sbin/gpt-build gcc32dbgpthr . $GPT_LOCATION/sbin/gpt-build gcc32dbg . $GPT_LOCATION/sbin/gpt-postinstall . copy in host{cert,key}.pem -> /etc/grid-security 9. post-gara-build extras. . cd /usr/local/src/gara-1.2.2/resource_manager/programs . compile execbin: . make -f mod_exec.compile . compile your flavor (AFS PTS or flatfile, mod_pts.acl) of mod_pts: . ./mod_pts.compile.afs *or* . ./mod_pts.compile.noafs . give yourself permissions in mod_pts.acl -- helps with debugging setup issues . if you want to turn on/off debugging in the diffserv manager, or if you want to en/disable PERMIS if you have a PERMIS-ified diffserv manager, you might edit the file: /resource_manager/programs/Makefile and play with the CFLAGS build variable. mine is: CFLAGS = -g -DGENERIC_RSL -DCLIENT_INFO -DCHECK_AUTHORIZATION -DDEBUG -DDEBUG_MOD_PTS -DPERMIS and -DDEBUG_PERMIS work, too, if you have the right diffserv_manager.c. 10. get your ~/.globus/ directory set up. . this can be finicky (e.g., with perms), but the globus.org docs help. . we have one that we copy around; email us for information. 11. set up /etc/grid-security (used by both globus and gara). . copy in host{cert,key}.pem -> /etc/grid-security . this involves having them made and, in our case, signed by the KDC. . copy in and/or otherwise set up /etc/grid-security/grid-mapfile . in our case, we have an allow-list of DNs who can run remote programs . make sure you have your signer's cert and signing policy (e.g., 078314af.0 and 078314af.signing_policy) in /etc/grid-security/certificates/ . they probably are/should (also) be in your ~/.globus/ directory. . repoint some conf file aliases in /etc/grid-security/ . come questo: sudo rm grid-security.conf sudo ln -s /etc/grid-security/certificates/grid-security.conf.078314af grid-security.conf sudo rm globus-user-ssl.conf sudo ln -s /etc/grid-security/certificates/globus-user-ssl.conf.078314af globus-user-ssl.conf sudo rm globus-host-ssl.conf sudo ln -s /etc/grid-security/certificates/globus-host-ssl.conf.078314af globus-host-ssl.conf . so, ours look like: globus-host-ssl.conf -> /etc/grid-security/certificates/globus-host-ssl.conf.078314af globus-user-ssl.conf -> /etc/grid-security/certificates/globus-user-ssl.conf.078314af grid-security.conf -> /etc/grid-security/certificates/grid-security.conf.078314af 12. edit /etc/services to contain the following line: gsigatekeeper 2119/tcp # Globus Gatekeeper 13. configure xinetd to start the globus_gatekeeper for you. . cd /etc/xinetd.d . vi globus-gatekeeper . fill the file with something like the following, then save: service gsigatekeeper { socket_type = stream protocol = tcp wait = no user = root server = /usr/local/src/globus-2.2.4/sbin/globus-gatekeeper server_args = -conf /usr/local/src/globus-2.2.4/etc/globus-gatekeeper.conf disable = no env = LD_LIBRARY_PATH=/usr/local/src/globus-2.2.4/lib } 14. tweak /usr/local/src/globus-2.2.4/etc/globus-gatekeeper.conf to fit your needs, if nec. 15. set up the globus gatekeeper gara service (quite a mouthful): . cd /usr/local/src/globus-2.2.4/etc/grid-services . vi gara-service . fill the file with something like the following, then save: stderr_log,local_cred - /usr/local/src/gara-1.2.2/bin/globus_gatekeeper_gara_service gara-service -d . as an aside, yes, that line means that the Globus 2.2.4 globus_gatekeeper runs with a globus_gatekeeper_gara_service that was compiled against Globus 2.4. 16. build the custom globus_gatekeeper (if necessary): . get the modified globus_gatekeeper.c (globus_gatekeeper.c.05_14_2003) from richterd@citi.umich.edu . cd /usr/local/src/globus-2.2.4/BUILD/globus_gatekeeper-2.1 . mv globus_gatekeeper.c{,_ORIG} . cp globus_gatekeeper.c . vi Makefile . edit the CFLAGS variable to contain -DCLIENT_INFO; our line reads: CFLAGS = -g -Wall -DCLIENT_INFO . make . make install 17. okay, try to start the gatekeeper: . sudo /etc/init.d/xinetd restart . netstat -na | less (look for port 2119) 18. test local globus stuff (globusrun FROM l99 back TO l99): . kinit . kx509 . kxlist -p (may have to set up env..) . source $GLOBUS_LOCATION/etc/globus-user-env.sh . minimal authentication test (done on host "l99"): . % globusrun -a -r l99 . you want to see: GRAM Authentication test successful . minimal remote execution test (again, done on host "l99"): . % globusrun -o -r l99 '&(executable=/bin/date)' . you want to see the normal output from /bin/date. 19. test the whole deal (in this case, via the webserver venice.citi.umich.edu, which is configured to initiate remote network tests): . first, have xinetd start the globus_gatekeeper. . next, you need to start a diffserv_manager: . cd /usr/local/src/gara-1.2.2/bin . mkdir -p /tmp/logs . chmod ugo+w /tmp/logs . sudo diffserv_manager . then, on some machine, run your pkcs11-savvy (XXX dmr: primer for this) browser, et al: . kinit . kx509 . mozilla & . .... having already added the libpkcs11.so security device to mozilla, of course