VFS leases

The Problem:
In order to use VFS leases to implement NFSv4 file delegations, we need to support breaking file leases on unlink, rename, chown, etc. To do so safely, we need a way to disable lease-granting for the duration of those operations in order to avoid races between the lease-breaking and lease-granting.

Currently, file leases are broken only when a file is opened or truncated. We need an approach that allows for lease-breaking without bad race conditions (e.g., the existing break-on-truncate and -open code has races), that won't cause lease-breakers to block while holding locks, and that won't add a lot of overhead to high-traffic code paths.

Samba also uses VFS leases to implement OPLOCK support; however, the existing lease subsystem doesn't actually break OPLOCK-leases all the times it "should" -- scroll to the bottom of the page for commentary and links.

Assuming we can disable lease-granting per-inode, how is it used?:

• when processing a lease-breaking operation, you're pretty much in one of two cases:
  • no locks (e.g. an i_mutex) are held at the point break_lease() is called, so a break is just:
    • 1. disable lease-granting on the affected inode(s)
    • 2. call break_lease() and possibly block while any outstanding leases are returned
    • 3. do the meat of the operation's processing.
    • 4. re-enable lease-granting.
  • if locks are held, they need to be dropped before the blocking break_lease() happens (bad case, it could take 90 seconds!), so the flow is roughly:
    • 1. disable lease-granting on the affected inode(s)
    • 2. attempt a non-blocking break_lease() -- if there are no leases present, just continue and skip to step #5.
    • 3. if you get -EWOULDBLOCK, drop the locks, call break_lease() and let it block
    • 4. .. when the break_lease() completes, reacquire the requisite locks and restart the operation
    • 5. do the meat of the operation's processing.
    • 6. re-enable lease-granting. the operation has completed in one pass either way.

• this approach eliminates races between lease-breaking and lease-granting operations.
  • NOTE: the existing sys_truncate() race is easily fixed in do_sys_truncate() by swapping get_write_access() and break_lease(). doesn't need any special lease-locking stuff.
  • NOTE: the existing open race is a bit harder. .. open_namei() would probably need to return with leases still locked on the inode in question, since e.g. i_writecount isn't incremented until nameidata_to_filp() later in the sys_open() path.

Summary of some possible solutions for disabling lease-granting per-inode:

• first approach: add something to the inode to tag it ("don't grant leases")
  • boo: adding a field to struct inode.
  • ?: hm, could we hide behind an #ifdef CONFIG_FATTY_INODES?

• second approach: create a new "lease-blocking" struct file_lock, add that to locklist
  • boo: if we had to allocate and free a bunch of blocking file_locks it'd be impossibly gross.
  • ?: could a small cache (10? 20?) of these mitigate the expense enough? how many inodes do we need locked at the same time, anyway?

• third approach: if there's a conflicting file_lock, tag it (with some refcount, say fl_lblocks - "number of parties preventing lease-granting") for the duration and make it persist until the end of the breaking-operation, then have the breaker decrement the refcount and free it when fl_lblocks is zero.
  • boo: changes struct file_lock, complicates locking and the freeing of leases, changes every use of things like vfs_unlink()

idea 1: abuse struct inode

• add a field to the struct inode -- "i_lblock" (1+ blocks/barriers to lease granting; grant only on 0)
• in, e.g., unlink, start with a new leases_lock_inode()
  • .. which does: lock_kernel, increment "i_lblock", unlock_kernel
• then unlink does its normal: break_lease(O_NONBLOCK), if(EWOULDBLOCK) drop locks, break_lease(), goto restart
• the use of the BKL in __setlease() (my version won't sleep) and leases_lock_inode() synchronizes the two..
• then in __setlease(), disallow granting leases if "i_lblock" is non-zero
• then, as long as things like unlink() use leases_lock_inode() to protect both break_lease() and the meat of the unlink, things should be OK.

• note that using existing "i-counts" checks isn't enough.
  • "i_count" can be safely incremented and thereby excluded write-lease-granting, but ..
  • "i_writecount" is what blocks read-lease-granting when the file is already opened for writing.
    • if we want to block read leases, let's try incrementing "i_writecount" ...
    • .. however, if a file's being executed, "i_writecount" needs to be negative to prevent writers.
    • .. and people should be able to get leases on files being executed.
    • .. so we couldn't safely use "i_writecount" to disable read-leases on executing binaries.

idea 2: create a lease-blocking struct file_lock

• similar in spirit to #1, "tag" the inode with a lease-blocking (or maybe ALL-other-lock-blocking) lock for the duration.
  • this is sorta like bruce's block-lock from his fair-queueing branch
• .. if we just add to the head of the locklist, there's no traversal penalty <shrug>
• .. maybe have a small cache of these preallocated?
• hrm, need to revisit this.

idea 3: abuse struct file_lock and a couple i_mutexes

• make both setlease() take both parent and child i_mutexes .. (more on this.)
• add a field to struct file_lock -- "fl_lblock"
• make __setlease() also check for "fl_lblocks" when trying to grant
• make break_lease() not free broken lease file_locks ...
  • instead, when break_lease() is called, traverse locklist and find all conflicters and increment their "fl_lblock"
  • then, breaker blocks while leases are returned
  • when breaker wakes up and returns
• make breaker free the conflicting locks when they're done with, e.g., unlink ..
• make setlease() lock both the parent-directory and the lease-target's i_mutexes .. why did we need both locked, again?
• boo: breaker might have to free a list of file_locks, since maybe there are 10 leases on that inode being broken

unfiled / maybe applicable to all?

• only grant leases with nlink == 1

samba

• samba uses VFS leases to implement OPLOCKs ("kernel oplocks", in their parlance)
  • [discussion of that]
  • hmm -- seems that samba is forced(?) to recommend that if files are accessed both by Windows and UNIXy clients, then OPLOCKs should just be disabled wholesale. Maybe we can help.
• samba breaks OPLOCKs when files are opened, it appears
  • hmm, microsoft thinks deletes should break oplocks.
    • [maybe linux should, too]
  • connectathon paper about Secure Share, which addresses related issues
    • [breaking on renames, deletes, etc] (pdf)
    • [USENIX paper on the same thing]
• would samba benefit from i-don't-break-my-own-leases semantics?

---