--- jsr166/src/jsr166y/ForkJoinPool.java 2010/07/24 20:28:18 1.60
+++ jsr166/src/jsr166y/ForkJoinPool.java 2010/08/11 18:45:12 1.61
@@ -156,25 +156,23 @@ public class ForkJoinPool extends Abstra
* ForkJoinWorkerThread.helpJoinTask tracks joining->stealing
* links to try to find such a task.
*
- * Compensating: Unless there are already enough live threads,
- * creating or or re-activating a spare thread to compensate
- * for the (blocked) joiner until it unblocks. Spares then
- * suspend at their next opportunity or eventually die if
- * unused for too long. See below and the internal
- * documentation for tryAwaitJoin for more details about
- * compensation rules.
+ * Compensating: Unless there are already enough live threads,
+ * method helpMaintainParallelism() may create or or
+ * re-activate a spare thread to compensate for blocked
+ * joiners until they unblock.
*
* Because the determining existence of conservatively safe
* helping targets, the availability of already-created spares,
* and the apparent need to create new spares are all racy and
- * require heuristic guidance, joins (in
- * ForkJoinWorkerThread.joinTask) interleave these options until
- * successful. Creating a new spare always succeeds, but also
- * increases application footprint, so we try to avoid it, within
- * reason.
+ * require heuristic guidance, we rely on multiple retries of
+ * each. Further, because it is impossible to keep exactly the
+ * target (parallelism) number of threads running at any given
+ * time, we allow compensation during joins to fail, and enlist
+ * all other threads to help out whenever they are not otherwise
+ * occupied (i.e., mainly in method preStep).
*
- * The ManagedBlocker extension API can't use helping so uses a
- * special version of compensation in method awaitBlocker.
+ * The ManagedBlocker extension API can't use helping so relies
+ * only on compensation in method awaitBlocker.
*
* The main throughput advantages of work-stealing stem from
* decentralized control -- workers mostly steal tasks from each
@@ -207,6 +205,19 @@ public class ForkJoinPool extends Abstra
* blocked workers. However, all other support code is set up to
* work with other policies.
*
+ * To ensure that we do not hold on to worker references that
+ * would prevent GC, ALL accesses to workers are via indices into
+ * the workers array (which is one source of some of the unusual
+ * code constructions here). In essence, the workers array serves
+ * as a WeakReference mechanism. Thus for example the event queue
+ * stores worker indices, not worker references. Access to the
+ * workers in associated methods (for example releaseEventWaiters)
+ * must both index-check and null-check the IDs. All such accesses
+ * ignore bad IDs by returning out early from what they are doing,
+ * since this can only be associated with shutdown, in which case
+ * it is OK to give up. On termination, we just clobber these
+ * data structures without trying to use them.
+ *
* 2. Bookkeeping for dynamically adding and removing workers. We
* aim to approximately maintain the given level of parallelism.
* When some workers are known to be blocked (on joins or via
@@ -248,28 +259,28 @@ public class ForkJoinPool extends Abstra
* workers that previously could not find a task to now find one:
* Submission of a new task to the pool, or another worker pushing
* a task onto a previously empty queue. (We also use this
- * mechanism for termination and reconfiguration actions that
- * require wakeups of idle workers). Each worker maintains its
- * last known event count, and blocks when a scan for work did not
- * find a task AND its lastEventCount matches the current
- * eventCount. Waiting idle workers are recorded in a variant of
- * Treiber stack headed by field eventWaiters which, when nonzero,
- * encodes the thread index and count awaited for by the worker
- * thread most recently calling eventSync. This thread in turn has
- * a record (field nextEventWaiter) for the next waiting worker.
- * In addition to allowing simpler decisions about need for
- * wakeup, the event count bits in eventWaiters serve the role of
- * tags to avoid ABA errors in Treiber stacks. To reduce delays
- * in task diffusion, workers not otherwise occupied may invoke
- * method releaseWaiters, that removes and signals (unparks)
- * workers not waiting on current count. To minimize task
- * production stalls associate with signalling, any worker pushing
- * a task on an empty queue invokes the weaker method signalWork,
- * that only releases idle workers until it detects interference
- * by other threads trying to release, and lets them take
- * over. The net effect is a tree-like diffusion of signals, where
- * released threads (and possibly others) help with unparks. To
- * further reduce contention effects a bit, failed CASes to
+ * mechanism for termination actions that require wakeups of idle
+ * workers). Each worker maintains its last known event count,
+ * and blocks when a scan for work did not find a task AND its
+ * lastEventCount matches the current eventCount. Waiting idle
+ * workers are recorded in a variant of Treiber stack headed by
+ * field eventWaiters which, when nonzero, encodes the thread
+ * index and count awaited for by the worker thread most recently
+ * calling eventSync. This thread in turn has a record (field
+ * nextEventWaiter) for the next waiting worker. In addition to
+ * allowing simpler decisions about need for wakeup, the event
+ * count bits in eventWaiters serve the role of tags to avoid ABA
+ * errors in Treiber stacks. To reduce delays in task diffusion,
+ * workers not otherwise occupied may invoke method
+ * releaseEventWaiters, that removes and signals (unparks) workers
+ * not waiting on current count. To reduce stalls, To minimize
+ * task production stalls associate with signalling, any worker
+ * pushing a task on an empty queue invokes the weaker method
+ * signalWork, that only releases idle workers until it detects
+ * interference by other threads trying to release, and lets them
+ * take over. The net effect is a tree-like diffusion of signals,
+ * where released threads (and possibly others) help with unparks.
+ * To further reduce contention effects a bit, failed CASes to
* increment field eventCount are tolerated without retries.
* Conceptually they are merged into the same event, which is OK
* when their only purpose is to enable workers to scan for work.
@@ -285,44 +296,43 @@ public class ForkJoinPool extends Abstra
* spare threads from normal "core" threads: On each call to
* preStep (the only point at which we can do this) a worker
* checks to see if there are now too many running workers, and if
- * so, suspends itself. Methods tryAwaitJoin and awaitBlocker
- * look for suspended threads to resume before considering
- * creating a new replacement. We don't need a special data
- * structure to maintain spares; simply scanning the workers array
- * looking for worker.isSuspended() is fine because the calling
- * thread is otherwise not doing anything useful anyway; we are at
- * least as happy if after locating a spare, the caller doesn't
- * actually block because the join is ready before we try to
- * adjust and compensate. Note that this is intrinsically racy.
- * One thread may become a spare at about the same time as another
- * is needlessly being created. We counteract this and related
- * slop in part by requiring resumed spares to immediately recheck
- * (in preStep) to see whether they they should re-suspend. The
- * only effective difference between "extra" and "core" threads is
- * that we allow the "extra" ones to time out and die if they are
- * not resumed within a keep-alive interval of a few seconds. This
- * is implemented mainly within ForkJoinWorkerThread, but requires
- * some coordination (isTrimmed() -- meaning killed while
- * suspended) to correctly maintain pool counts.
+ * so, suspends itself. Method helpMaintainParallelism looks for
+ * suspended threads to resume before considering creating a new
+ * replacement. The spares themselves are encoded on another
+ * variant of a Treiber Stack, headed at field "spareWaiters".
+ * Note that the use of spares is intrinsically racy. One thread
+ * may become a spare at about the same time as another is
+ * needlessly being created. We counteract this and related slop
+ * in part by requiring resumed spares to immediately recheck (in
+ * preStep) to see whether they they should re-suspend. To avoid
+ * long-term build-up of spares, the oldest spare (see
+ * ForkJoinWorkerThread.suspendAsSpare) occasionally wakes up if
+ * not signalled and calls tryTrimSpare, which uses two different
+ * thresholds: Always killing if the number of spares is greater
+ * that 25% of total, and killing others only at a slower rate
+ * (UNUSED_SPARE_TRIM_RATE_NANOS).
*
* 6. Deciding when to create new workers. The main dynamic
- * control in this class is deciding when to create extra threads,
- * in methods awaitJoin and awaitBlocker. We always need to create
- * one when the number of running threads would become zero and
- * all workers are busy. However, this is not easy to detect
- * reliably in the presence of transients so we use retries and
- * allow slack (in tryAwaitJoin) to reduce false alarms. These
- * effectively reduce churn at the price of systematically
- * undershooting target parallelism when many threads are blocked.
- * However, biasing toward undeshooting partially compensates for
- * the above mechanics to suspend extra threads, that normally
- * lead to overshoot because we can only suspend workers
- * in-between top-level actions. It also better copes with the
- * fact that some of the methods in this class tend to never
- * become compiled (but are interpreted), so some components of
- * the entire set of controls might execute many times faster than
- * others. And similarly for cases where the apparent lack of work
- * is just due to GC stalls and other transient system activity.
+ * control in this class is deciding when to create extra threads
+ * in method helpMaintainParallelism. We would like to keep
+ * exactly #parallelism threads running, which is an impossble
+ * task. We always need to create one when the number of running
+ * threads would become zero and all workers are busy. Beyond
+ * this, we must rely on heuristics that work well in the the
+ * presence of transients phenomena such as GC stalls, dynamic
+ * compilation, and wake-up lags. These transients are extremely
+ * common -- we are normally trying to fully saturate the CPUs on
+ * a machine, so almost any activity other than running tasks
+ * impedes accuracy. Our main defense is to allow some slack in
+ * creation thresholds, using rules that reflect the fact that the
+ * more threads we have running, the more likely that we are
+ * underestimating the number running threads. The rules also
+ * better cope with the fact that some of the methods in this
+ * class tend to never become compiled (but are interpreted), so
+ * some components of the entire set of controls might execute 100
+ * times faster than others. And similarly for cases where the
+ * apparent lack of work is just due to GC stalls and other
+ * transient system activity.
*
* Beware that there is a lot of representation-level coupling
* among classes ForkJoinPool, ForkJoinWorkerThread, and
@@ -335,11 +345,13 @@ public class ForkJoinPool extends Abstra
*
* Style notes: There are lots of inline assignments (of form
* "while ((local = field) != 0)") which are usually the simplest
- * way to ensure read orderings. Also several occurrences of the
- * unusual "do {} while(!cas...)" which is the simplest way to
- * force an update of a CAS'ed variable. There are also other
- * coding oddities that help some methods perform reasonably even
- * when interpreted (not compiled), at the expense of messiness.
+ * way to ensure the required read orderings (which are sometimes
+ * critical). Also several occurrences of the unusual "do {}
+ * while(!cas...)" which is the simplest way to force an update of
+ * a CAS'ed variable. There are also other coding oddities that
+ * help some methods perform reasonably even when interpreted (not
+ * compiled), at the expense of some messy constructions that
+ * reduce byte code counts.
*
* The order of declarations in this file is: (1) statics (2)
* fields (along with constants used when unpacking some of them)
@@ -407,10 +419,11 @@ public class ForkJoinPool extends Abstra
new AtomicInteger();
/**
- * Absolute bound for parallelism level. Twice this number must
- * fit into a 16bit field to enable word-packing for some counts.
+ * Absolute bound for parallelism level. Twice this number plus
+ * one (i.e., 0xfff) must fit into a 16bit field to enable
+ * word-packing for some counts and indices.
*/
- private static final int MAX_THREADS = 0x7fff;
+ private static final int MAX_WORKERS = 0x7fff;
/**
* Array holding all worker threads in the pool. Array size must
@@ -450,25 +463,47 @@ public class ForkJoinPool extends Abstra
private volatile long stealCount;
/**
+ * The last nanoTime that a spare thread was trimmed
+ */
+ private volatile long trimTime;
+
+ /**
+ * The rate at which to trim unused spares
+ */
+ static final long UNUSED_SPARE_TRIM_RATE_NANOS =
+ 1000L * 1000L * 1000L; // 1 sec
+
+ /**
* Encoded record of top of treiber stack of threads waiting for
* events. The top 32 bits contain the count being waited for. The
- * bottom word contains one plus the pool index of waiting worker
- * thread.
+ * bottom 16 bits contains one plus the pool index of waiting
+ * worker thread. (Bits 16-31 are unused.)
*/
private volatile long eventWaiters;
private static final int EVENT_COUNT_SHIFT = 32;
- private static final long WAITER_ID_MASK = (1L << EVENT_COUNT_SHIFT)-1L;
+ private static final long WAITER_ID_MASK = (1L << 16) - 1L;
/**
* A counter for events that may wake up worker threads:
* - Submission of a new task to the pool
* - A worker pushing a task on an empty queue
- * - termination and reconfiguration
+ * - termination
*/
private volatile int eventCount;
/**
+ * Encoded record of top of treiber stack of spare threads waiting
+ * for resumption. The top 16 bits contain an arbitrary count to
+ * avoid ABA effects. The bottom 16bits contains one plus the pool
+ * index of waiting worker thread.
+ */
+ private volatile int spareWaiters;
+
+ private static final int SPARE_COUNT_SHIFT = 16;
+ private static final int SPARE_ID_MASK = (1 << 16) - 1;
+
+ /**
* Lifecycle control. The low word contains the number of workers
* that are (probably) executing tasks. This value is atomically
* incremented before a worker gets a task to run, and decremented
@@ -529,11 +564,12 @@ public class ForkJoinPool extends Abstra
*/
private final int poolNumber;
+
// Utilities for CASing fields. Note that several of these
// are manually inlined by callers
/**
- * Increments running count. Also used by ForkJoinTask.
+ * Increments running count part of workerCounts
*/
final void incrementRunningCount() {
int c;
@@ -554,12 +590,33 @@ public class ForkJoinPool extends Abstra
}
/**
- * Tries to increment running count
+ * Forces decrement of encoded workerCounts, awaiting nonzero if
+ * (rarely) necessary when other count updates lag.
+ *
+ * @param dr -- either zero or ONE_RUNNING
+ * @param dt == either zero or ONE_TOTAL
*/
- final boolean tryIncrementRunningCount() {
- int wc;
- return UNSAFE.compareAndSwapInt(this, workerCountsOffset,
- wc = workerCounts, wc + ONE_RUNNING);
+ private void decrementWorkerCounts(int dr, int dt) {
+ for (;;) {
+ int wc = workerCounts;
+ if (wc == 0 && (runState & TERMINATED) != 0)
+ return; // lagging termination on a backout
+ if ((wc & RUNNING_COUNT_MASK) - dr < 0 ||
+ (wc >>> TOTAL_COUNT_SHIFT) - dt < 0)
+ Thread.yield();
+ if (UNSAFE.compareAndSwapInt(this, workerCountsOffset,
+ wc, wc - (dr + dt)))
+ return;
+ }
+ }
+
+ /**
+ * Increments event count
+ */
+ private void advanceEventCount() {
+ int c;
+ do {} while(!UNSAFE.compareAndSwapInt(this, eventCountOffset,
+ c = eventCount, c+1));
}
/**
@@ -610,12 +667,12 @@ public class ForkJoinPool extends Abstra
lock.lock();
try {
ForkJoinWorkerThread[] ws = workers;
- int nws = ws.length;
- if (k < 0 || k >= nws || ws[k] != null) {
- for (k = 0; k < nws && ws[k] != null; ++k)
+ int n = ws.length;
+ if (k < 0 || k >= n || ws[k] != null) {
+ for (k = 0; k < n && ws[k] != null; ++k)
;
- if (k == nws)
- ws = Arrays.copyOf(ws, nws << 1);
+ if (k == n)
+ ws = Arrays.copyOf(ws, n << 1);
}
ws[k] = w;
workers = ws; // volatile array write ensures slot visibility
@@ -648,401 +705,341 @@ public class ForkJoinPool extends Abstra
* Tries to create and add new worker. Assumes that worker counts
* are already updated to accommodate the worker, so adjusts on
* failure.
- *
- * @return new worker or null if creation failed
*/
- private ForkJoinWorkerThread addWorker() {
+ private void addWorker() {
ForkJoinWorkerThread w = null;
try {
w = factory.newThread(this);
} finally { // Adjust on either null or exceptional factory return
- if (w == null)
- onWorkerCreationFailure();
+ if (w == null) {
+ decrementWorkerCounts(ONE_RUNNING, ONE_TOTAL);
+ tryTerminate(false); // in case of failure during shutdown
+ }
}
if (w != null)
w.start(recordWorker(w), ueh);
- return w;
- }
-
- /**
- * Adjusts counts upon failure to create worker
- */
- private void onWorkerCreationFailure() {
- for (;;) {
- int wc = workerCounts;
- int rc = wc & RUNNING_COUNT_MASK;
- int tc = wc >>> TOTAL_COUNT_SHIFT;
- if (rc == 0 || wc == 0)
- Thread.yield(); // must wait for other counts to settle
- else if (UNSAFE.compareAndSwapInt(this, workerCountsOffset, wc,
- wc - (ONE_RUNNING|ONE_TOTAL)))
- break;
- }
- tryTerminate(false); // in case of failure during shutdown
- }
-
- /**
- * Creates enough total workers to establish target parallelism,
- * giving up if terminating or addWorker fails
- */
- private void ensureEnoughTotalWorkers() {
- int wc;
- while (((wc = workerCounts) >>> TOTAL_COUNT_SHIFT) < parallelism &&
- runState < TERMINATING) {
- if ((UNSAFE.compareAndSwapInt(this, workerCountsOffset,
- wc, wc + (ONE_RUNNING|ONE_TOTAL)) &&
- addWorker() == null))
- break;
- }
}
/**
* Final callback from terminating worker. Removes record of
* worker from array, and adjusts counts. If pool is shutting
- * down, tries to complete terminatation, else possibly replaces
- * the worker.
+ * down, tries to complete terminatation.
*
* @param w the worker
*/
final void workerTerminated(ForkJoinWorkerThread w) {
- if (w.active) { // force inactive
- w.active = false;
- do {} while (!tryDecrementActiveCount());
- }
forgetWorker(w);
-
- // Decrement total count, and if was running, running count
- // Spin (waiting for other updates) if either would be negative
- int nr = w.isTrimmed() ? 0 : ONE_RUNNING;
- int unit = ONE_TOTAL + nr;
- for (;;) {
- int wc = workerCounts;
- int rc = wc & RUNNING_COUNT_MASK;
- int tc = wc >>> TOTAL_COUNT_SHIFT;
- if (rc - nr < 0 || tc == 0)
- Thread.yield(); // back off if waiting for other updates
- else if (UNSAFE.compareAndSwapInt(this, workerCountsOffset,
- wc, wc - unit))
- break;
- }
-
- accumulateStealCount(w); // collect final count
- if (!tryTerminate(false))
- ensureEnoughTotalWorkers();
+ decrementWorkerCounts(w.isTrimmed()? 0 : ONE_RUNNING, ONE_TOTAL);
+ while (w.stealCount != 0) // collect final count
+ tryAccumulateStealCount(w);
+ tryTerminate(false);
}
// Waiting for and signalling events
/**
* Releases workers blocked on a count not equal to current count.
- * @return true if any released
+ * Normally called after precheck that eventWaiters isn't zero to
+ * avoid wasted array checks.
+ *
+ * @param signalling true if caller is a signalling worker so can
+ * exit upon (conservatively) detected contention by other threads
+ * who will continue to release
*/
- private void releaseWaiters() {
- long top;
- while ((top = eventWaiters) != 0L) {
- ForkJoinWorkerThread[] ws = workers;
- int n = ws.length;
- for (;;) {
- int i = ((int)(top & WAITER_ID_MASK)) - 1;
- int e = (int)(top >>> EVENT_COUNT_SHIFT);
- if (i < 0 || e == eventCount)
- return;
- ForkJoinWorkerThread w;
- if (i < n && (w = ws[i]) != null &&
- UNSAFE.compareAndSwapLong(this, eventWaitersOffset,
- top, w.nextWaiter)) {
- LockSupport.unpark(w);
- top = eventWaiters;
- }
- else
- break; // possibly stale; reread
- }
+ private void releaseEventWaiters(boolean signalling) {
+ ForkJoinWorkerThread[] ws = workers;
+ int n = ws.length;
+ long h; // head of stack
+ ForkJoinWorkerThread w; int id, ec;
+ while ((id = ((int)((h = eventWaiters) & WAITER_ID_MASK)) - 1) >= 0 &&
+ (int)(h >>> EVENT_COUNT_SHIFT) != (ec = eventCount) &&
+ id < n && (w = ws[id]) != null) {
+ if (UNSAFE.compareAndSwapLong(this, eventWaitersOffset,
+ h, h = w.nextWaiter))
+ LockSupport.unpark(w);
+ if (signalling && (eventCount != ec || eventWaiters != h))
+ break;
}
}
/**
- * Ensures eventCount on exit is different (mod 2^32) than on
- * entry and wakes up all waiters
- */
- private void signalEvent() {
- int c;
- do {} while (!UNSAFE.compareAndSwapInt(this, eventCountOffset,
- c = eventCount, c+1));
- releaseWaiters();
- }
-
- /**
- * Advances eventCount and releases waiters until interference by
- * other releasing threads is detected.
+ * Tries to advance eventCount and releases waiters. Called only
+ * from workers.
*/
final void signalWork() {
- int c;
- UNSAFE.compareAndSwapInt(this, eventCountOffset, c=eventCount, c+1);
- long top;
- while ((top = eventWaiters) != 0L) {
- int ec = eventCount;
- ForkJoinWorkerThread[] ws = workers;
- int n = ws.length;
- for (;;) {
- int i = ((int)(top & WAITER_ID_MASK)) - 1;
- int e = (int)(top >>> EVENT_COUNT_SHIFT);
- if (i < 0 || e == ec)
- return;
- ForkJoinWorkerThread w;
- if (i < n && (w = ws[i]) != null &&
- UNSAFE.compareAndSwapLong(this, eventWaitersOffset,
- top, top = w.nextWaiter)) {
- LockSupport.unpark(w);
- if (top != eventWaiters) // let someone else take over
- return;
- }
- else
- break; // possibly stale; reread
- }
- }
+ int c; // try to increment event count -- CAS failure OK
+ UNSAFE.compareAndSwapInt(this, eventCountOffset, c = eventCount, c+1);
+ if (eventWaiters != 0L)
+ releaseEventWaiters(true);
}
/**
- * Blockss worker until terminating or event count
+ * Blocks worker until terminating or event count
* advances from last value held by worker
*
* @param w the calling worker thread
*/
private void eventSync(ForkJoinWorkerThread w) {
int wec = w.lastEventCount;
- long nextTop = (((long)wec << EVENT_COUNT_SHIFT) |
- ((long)(w.poolIndex + 1)));
- long top;
+ long nh = (((long)wec) << EVENT_COUNT_SHIFT) | ((long)(w.poolIndex+1));
+ long h;
while ((runState < SHUTDOWN || !tryTerminate(false)) &&
- (((int)(top = eventWaiters) & WAITER_ID_MASK) == 0 ||
- (int)(top >>> EVENT_COUNT_SHIFT) == wec) &&
+ ((h = eventWaiters) == 0L ||
+ (int)(h >>> EVENT_COUNT_SHIFT) == wec) &&
eventCount == wec) {
if (UNSAFE.compareAndSwapLong(this, eventWaitersOffset,
- w.nextWaiter = top, nextTop)) {
- accumulateStealCount(w); // transfer steals while idle
- Thread.interrupted(); // clear/ignore interrupt
- while (eventCount == wec)
- w.doPark();
+ w.nextWaiter = h, nh)) {
+ while (runState < TERMINATING && eventCount == wec) {
+ if (!tryAccumulateStealCount(w)) // transfer while idle
+ continue;
+ Thread.interrupted(); // clear/ignore interrupt
+ if (eventCount != wec)
+ break;
+ LockSupport.park(w);
+ }
break;
}
}
w.lastEventCount = eventCount;
}
+ // Maintaining spares
+
+ /**
+ * Pushes worker onto the spare stack
+ */
+ final void pushSpare(ForkJoinWorkerThread w) {
+ int ns = (++w.spareCount << SPARE_COUNT_SHIFT) | (w.poolIndex+1);
+ do {} while (!UNSAFE.compareAndSwapInt(this, spareWaitersOffset,
+ w.nextSpare = spareWaiters,ns));
+ }
+
+ /**
+ * Tries (once) to resume a spare if running count is less than
+ * target parallelism. Fails on contention or stale workers.
+ */
+ private void tryResumeSpare() {
+ int sw, id;
+ ForkJoinWorkerThread w;
+ ForkJoinWorkerThread[] ws;
+ if ((id = ((sw = spareWaiters) & SPARE_ID_MASK) - 1) >= 0 &&
+ id < (ws = workers).length && (w = ws[id]) != null &&
+ (workerCounts & RUNNING_COUNT_MASK) < parallelism &&
+ eventWaiters == 0L &&
+ spareWaiters == sw &&
+ UNSAFE.compareAndSwapInt(this, spareWaitersOffset,
+ sw, w.nextSpare) &&
+ w.tryUnsuspend()) {
+ int c; // try increment; if contended, finish after unpark
+ boolean inc = UNSAFE.compareAndSwapInt(this, workerCountsOffset,
+ c = workerCounts,
+ c + ONE_RUNNING);
+ LockSupport.unpark(w);
+ if (!inc) {
+ do {} while(!UNSAFE.compareAndSwapInt(this, workerCountsOffset,
+ c = workerCounts,
+ c + ONE_RUNNING));
+ }
+ }
+ }
+
+ /**
+ * Callback from oldest spare occasionally waking up. Tries
+ * (once) to shutdown a spare if more than 25% spare overage, or
+ * if UNUSED_SPARE_TRIM_RATE_NANOS have elapsed and there are at
+ * least #parallelism running threads. Note that we don't need CAS
+ * or locks here because the method is called only from the oldest
+ * suspended spare occasionally waking (and even misfires are OK).
+ *
+ * @param now the wake up nanoTime of caller
+ */
+ final void tryTrimSpare(long now) {
+ long lastTrim = trimTime;
+ trimTime = now;
+ helpMaintainParallelism(); // first, help wake up any needed spares
+ int sw, id;
+ ForkJoinWorkerThread w;
+ ForkJoinWorkerThread[] ws;
+ int pc = parallelism;
+ int wc = workerCounts;
+ if ((wc & RUNNING_COUNT_MASK) >= pc &&
+ (((wc >>> TOTAL_COUNT_SHIFT) - pc) > (pc >>> 2) + 1 ||// approx 25%
+ now - lastTrim >= UNUSED_SPARE_TRIM_RATE_NANOS) &&
+ (id = ((sw = spareWaiters) & SPARE_ID_MASK) - 1) >= 0 &&
+ id < (ws = workers).length && (w = ws[id]) != null &&
+ UNSAFE.compareAndSwapInt(this, spareWaitersOffset,
+ sw, w.nextSpare))
+ w.shutdown(false);
+ }
+
+ /**
+ * Does at most one of:
+ *
+ * 1. Help wake up existing workers waiting for work via
+ * releaseEventWaiters. (If any exist, then it probably doesn't
+ * matter right now if under target parallelism level.)
+ *
+ * 2. If below parallelism level and a spare exists, try (once)
+ * to resume it via tryResumeSpare.
+ *
+ * 3. If neither of the above, tries (once) to add a new
+ * worker if either there are not enough total, or if all
+ * existing workers are busy, there are either no running
+ * workers or the deficit is at least twice the surplus.
+ */
+ private void helpMaintainParallelism() {
+ // uglified to work better when not compiled
+ int pc, wc, rc, tc, rs; long h;
+ if ((h = eventWaiters) != 0L) {
+ if ((int)(h >>> EVENT_COUNT_SHIFT) != eventCount)
+ releaseEventWaiters(false); // avoid useless call
+ }
+ else if ((pc = parallelism) >
+ (rc = ((wc = workerCounts) & RUNNING_COUNT_MASK))) {
+ if (spareWaiters != 0)
+ tryResumeSpare();
+ else if ((rs = runState) < TERMINATING &&
+ ((tc = wc >>> TOTAL_COUNT_SHIFT) < pc ||
+ (tc == (rs & ACTIVE_COUNT_MASK) && // all busy
+ (rc == 0 || // must add
+ rc < pc - ((tc - pc) << 1)) && // within slack
+ tc < MAX_WORKERS && runState == rs)) && // recheck busy
+ workerCounts == wc &&
+ UNSAFE.compareAndSwapInt(this, workerCountsOffset, wc,
+ wc + (ONE_RUNNING|ONE_TOTAL)))
+ addWorker();
+ }
+ }
+
/**
* Callback from workers invoked upon each top-level action (i.e.,
* stealing a task or taking a submission and running
- * it). Performs one or both of the following:
+ * it). Performs one or more of the following:
+ *
+ * 1. If the worker cannot find work (misses > 0), updates its
+ * active status to inactive and updates activeCount unless
+ * this is the first miss and there is contention, in which
+ * case it may try again (either in this or a subsequent
+ * call).
+ *
+ * 2. If there are at least 2 misses, awaits the next task event
+ * via eventSync
+ *
+ * 3. If there are too many running threads, suspends this worker
+ * (first forcing inactivation if necessary). If it is not
+ * needed, it may be killed while suspended via
+ * tryTrimSpare. Otherwise, upon resume it rechecks to make
+ * sure that it is still needed.
*
- * * If the worker cannot find work, updates its active status to
- * inactive and updates activeCount unless there is contention, in
- * which case it may try again (either in this or a subsequent
- * call). Additionally, awaits the next task event and/or helps
- * wake up other releasable waiters.
- *
- * * If there are too many running threads, suspends this worker
- * (first forcing inactivation if necessary). If it is not
- * resumed before a keepAlive elapses, the worker may be "trimmed"
- * -- killed while suspended within suspendAsSpare. Otherwise,
- * upon resume it rechecks to make sure that it is still needed.
+ * 4. Helps release and/or reactivate other workers via
+ * helpMaintainParallelism
*
* @param w the worker
- * @param retries the number of scans by caller failing to find work
- * find any (in which case it may block waiting for work).
+ * @param misses the number of scans by caller failing to find work
+ * (saturating at 2 just to avoid wraparound)
*/
- final void preStep(ForkJoinWorkerThread w, int retries) {
+ final void preStep(ForkJoinWorkerThread w, int misses) {
boolean active = w.active;
- boolean inactivate = active && retries > 0;
+ int pc = parallelism;
for (;;) {
- int rs, wc;
- if (inactivate &&
- UNSAFE.compareAndSwapInt(this, runStateOffset,
- rs = runState, rs - ONE_ACTIVE))
- inactivate = active = w.active = false;
- if (((wc = workerCounts) & RUNNING_COUNT_MASK) <= parallelism) {
- if (retries > 0) {
- if (retries > 1 && !active)
- eventSync(w);
- releaseWaiters();
- }
- break;
+ int wc = workerCounts;
+ int rc = wc & RUNNING_COUNT_MASK;
+ if (active && (misses > 0 || rc > pc)) {
+ int rs; // try inactivate
+ if (UNSAFE.compareAndSwapInt(this, runStateOffset,
+ rs = runState, rs - ONE_ACTIVE))
+ active = w.active = false;
+ else if (misses > 1 || rc > pc ||
+ (rs & ACTIVE_COUNT_MASK) >= pc)
+ continue; // force inactivate
+ }
+ if (misses > 1) {
+ misses = 0; // don't re-sync
+ eventSync(w); // continue loop to recheck rc
+ }
+ else if (rc > pc) {
+ if (workerCounts == wc && // try to suspend as spare
+ UNSAFE.compareAndSwapInt(this, workerCountsOffset,
+ wc, wc - ONE_RUNNING) &&
+ !w.suspendAsSpare()) // false if killed
+ break;
}
- if (!(inactivate |= active) && // must inactivate to suspend
- UNSAFE.compareAndSwapInt(this, workerCountsOffset,
- wc, wc - ONE_RUNNING) &&
- !w.suspendAsSpare()) // false if trimmed
+ else {
+ if (rc < pc || eventWaiters != 0L)
+ helpMaintainParallelism();
break;
+ }
}
}
/**
- * Awaits join of the given task if enough threads, or can resume
- * or create a spare. Fails (in which case the given task might
- * not be done) upon contention or lack of decision about
- * blocking.
- *
- * We allow blocking if:
- *
- * 1. There would still be at least as many running threads as
- * parallelism level if this thread blocks.
- *
- * 2. A spare is resumed to replace this worker. We tolerate
- * races in the decision to replace when a spare is found.
- * This may release too many, but if so, the superfluous ones
- * will re-suspend via preStep().
- *
- * 3. After #spares repeated retries, there are fewer than #spare
- * threads not running. We allow this slack to avoid hysteresis
- * and as a hedge against lag/uncertainty of running count
- * estimates when signalling or unblocking stalls.
- *
- * 4. All existing workers are busy (as rechecked via #spares
- * repeated retries by caller) and a new spare is created.
- *
- * If none of the above hold, we escape out by re-incrementing
- * count and returning to caller, which can retry later.
+ * Helps and/or blocks awaiting join of the given task.
+ * Alternates between helpJoinTask() and helpMaintainParallelism()
+ * as many times as there is a deficit in running count (or longer
+ * if running count would become zero), then blocks if task still
+ * not done.
*
* @param joinMe the task to join
- * @param retries the number of calls to this method for this join
*/
- final void tryAwaitJoin(ForkJoinTask joinMe, int retries) {
- int pc = parallelism;
- boolean running = true; // false when running count decremented
- outer:while (joinMe.status >= 0) {
- int wc = workerCounts;
- int rc = wc & RUNNING_COUNT_MASK;
- int tc = wc >>> TOTAL_COUNT_SHIFT;
- if (running) { // replace with spare or decrement count
- if (rc <= pc && tc > pc &&
- (retries > 0 || tc > (runState & ACTIVE_COUNT_MASK))) {
- ForkJoinWorkerThread[] ws = workers; // search for spare
- int nws = ws.length;
- for (int i = 0; i < nws; ++i) {
- ForkJoinWorkerThread w = ws[i];
- if (w != null && w.isSuspended()) {
- if ((workerCounts & RUNNING_COUNT_MASK) > pc)
- continue outer;
- if (joinMe.status < 0)
- break outer;
- if (w.tryResumeSpare()) {
- running = false;
- break outer;
- }
- continue outer; // rescan on failure to resume
- }
- }
- }
- if ((rc <= pc && (rc == 0 || --retries < 0)) || // no retry
- joinMe.status < 0)
- break;
- if (workerCounts == wc &&
- UNSAFE.compareAndSwapInt(this, workerCountsOffset,
- wc, wc - ONE_RUNNING))
- running = false;
+ final void awaitJoin(ForkJoinTask joinMe, ForkJoinWorkerThread worker) {
+ int threshold = parallelism; // descend blocking thresholds
+ while (joinMe.status >= 0) {
+ boolean block; int wc;
+ worker.helpJoinTask(joinMe);
+ if (joinMe.status < 0)
+ break;
+ if (((wc = workerCounts) & RUNNING_COUNT_MASK) <= threshold) {
+ if (threshold > 0)
+ --threshold;
+ else
+ advanceEventCount(); // force release
+ block = false;
}
- else { // allow blocking if enough threads
- int sc = tc - pc + 1; // = spares, plus the one to add
- if (sc > 0 && rc > 0 && rc >= pc - sc && rc > pc - retries)
- break;
- if (--retries > sc && tc < MAX_THREADS &&
- tc == (runState & ACTIVE_COUNT_MASK) &&
- workerCounts == wc &&
- UNSAFE.compareAndSwapInt(this, workerCountsOffset, wc,
- wc + (ONE_RUNNING|ONE_TOTAL))) {
- addWorker();
- break;
- }
- if (workerCounts == wc &&
- UNSAFE.compareAndSwapInt (this, workerCountsOffset,
- wc, wc + ONE_RUNNING)) {
- running = true; // back out; allow retry
- break;
- }
+ else
+ block = UNSAFE.compareAndSwapInt(this, workerCountsOffset,
+ wc, wc - ONE_RUNNING);
+ helpMaintainParallelism();
+ if (block) {
+ int c;
+ joinMe.internalAwaitDone();
+ do {} while (!UNSAFE.compareAndSwapInt
+ (this, workerCountsOffset,
+ c = workerCounts, c + ONE_RUNNING));
+ break;
}
}
- if (!running) { // can block
- int c; // to inline incrementRunningCount
- joinMe.internalAwaitDone();
- do {} while (!UNSAFE.compareAndSwapInt
- (this, workerCountsOffset,
- c = workerCounts, c + ONE_RUNNING));
- }
}
/**
- * Same idea as (and shares many code snippets with) tryAwaitJoin,
- * but self-contained because there are no caller retries.
- * TODO: Rework to use simpler API.
+ * Same idea as awaitJoin, but no helping
*/
final void awaitBlocker(ManagedBlocker blocker)
throws InterruptedException {
- int pc = parallelism;
- boolean running = true;
- int retries = 0;
- boolean done;
- outer:while (!(done = blocker.isReleasable())) {
- int wc = workerCounts;
- int rc = wc & RUNNING_COUNT_MASK;
- int tc = wc >>> TOTAL_COUNT_SHIFT;
- if (running) {
- if (rc <= pc && tc > pc &&
- (retries > 0 || tc > (runState & ACTIVE_COUNT_MASK))) {
- ForkJoinWorkerThread[] ws = workers;
- int nws = ws.length;
- for (int i = 0; i < nws; ++i) {
- ForkJoinWorkerThread w = ws[i];
- if (w != null && w.isSuspended()) {
- if ((workerCounts & RUNNING_COUNT_MASK) > pc)
- continue outer;
- if (done = blocker.isReleasable())
- break outer;
- if (w.tryResumeSpare()) {
- running = false;
- break outer;
- }
- continue outer;
- }
- }
- if (done = blocker.isReleasable())
- break;
- }
- if (rc > 0 && workerCounts == wc &&
- UNSAFE.compareAndSwapInt(this, workerCountsOffset,
- wc, wc - ONE_RUNNING)) {
- running = false;
- if (rc > pc)
- break;
- }
+ int threshold = parallelism;
+ while (!blocker.isReleasable()) {
+ boolean block; int wc;
+ if (((wc = workerCounts) & RUNNING_COUNT_MASK) <= threshold) {
+ if (threshold > 0)
+ --threshold;
+ else
+ advanceEventCount();
+ block = false;
}
- else if (rc >= pc)
- break;
- else if (tc < MAX_THREADS &&
- tc == (runState & ACTIVE_COUNT_MASK) &&
- workerCounts == wc &&
- UNSAFE.compareAndSwapInt(this, workerCountsOffset, wc,
- wc + (ONE_RUNNING|ONE_TOTAL))) {
- addWorker();
+ else
+ block = UNSAFE.compareAndSwapInt(this, workerCountsOffset,
+ wc, wc - ONE_RUNNING);
+ helpMaintainParallelism();
+ if (block) {
+ try {
+ do {} while (!blocker.isReleasable() && !blocker.block());
+ } finally {
+ int c;
+ do {} while (!UNSAFE.compareAndSwapInt
+ (this, workerCountsOffset,
+ c = workerCounts, c + ONE_RUNNING));
+ }
break;
}
- else if (workerCounts == wc &&
- UNSAFE.compareAndSwapInt (this, workerCountsOffset,
- wc, wc + ONE_RUNNING)) {
- Thread.yield();
- ++retries;
- running = true; // allow rescan
- }
- }
-
- try {
- if (!done)
- do {} while (!blocker.isReleasable() && !blocker.block());
- } finally {
- if (!running) {
- int c;
- do {} while (!UNSAFE.compareAndSwapInt
- (this, workerCountsOffset,
- c = workerCounts, c + ONE_RUNNING));
- }
}
}
@@ -1074,14 +1071,37 @@ public class ForkJoinPool extends Abstra
/**
* Actions on transition to TERMINATING
+ *
+ * Runs up to four passes through workers: (0) shutting down each
+ * quietly (without waking up if parked) to quickly spread
+ * notifications without unnecessary bouncing around event queues
+ * etc (1) wake up and help cancel tasks (2) interrupt (3) mop up
+ * races with interrupted workers
*/
private void startTerminating() {
- for (int i = 0; i < 2; ++i) { // twice to mop up newly created workers
- cancelSubmissions();
- shutdownWorkers();
- cancelWorkerTasks();
- signalEvent();
- interruptWorkers();
+ cancelSubmissions();
+ for (int passes = 0; passes < 4 && workerCounts != 0; ++passes) {
+ advanceEventCount();
+ eventWaiters = 0L; // clobber lists
+ spareWaiters = 0;
+ ForkJoinWorkerThread[] ws = workers;
+ int n = ws.length;
+ for (int i = 0; i < n; ++i) {
+ ForkJoinWorkerThread w = ws[i];
+ if (w != null) {
+ w.shutdown(true);
+ if (passes > 0 && !w.isTerminated()) {
+ w.cancelTasks();
+ LockSupport.unpark(w);
+ if (passes > 1) {
+ try {
+ w.interrupt();
+ } catch (SecurityException ignore) {
+ }
+ }
+ }
+ }
+ }
}
}
@@ -1098,50 +1118,6 @@ public class ForkJoinPool extends Abstra
}
}
- /**
- * Sets all worker run states to at least shutdown,
- * also resuming suspended workers
- */
- private void shutdownWorkers() {
- ForkJoinWorkerThread[] ws = workers;
- int nws = ws.length;
- for (int i = 0; i < nws; ++i) {
- ForkJoinWorkerThread w = ws[i];
- if (w != null)
- w.shutdown();
- }
- }
-
- /**
- * Clears out and cancels all locally queued tasks
- */
- private void cancelWorkerTasks() {
- ForkJoinWorkerThread[] ws = workers;
- int nws = ws.length;
- for (int i = 0; i < nws; ++i) {
- ForkJoinWorkerThread w = ws[i];
- if (w != null)
- w.cancelTasks();
- }
- }
-
- /**
- * Unsticks all workers blocked on joins etc
- */
- private void interruptWorkers() {
- ForkJoinWorkerThread[] ws = workers;
- int nws = ws.length;
- for (int i = 0; i < nws; ++i) {
- ForkJoinWorkerThread w = ws[i];
- if (w != null && !w.isTerminated()) {
- try {
- w.interrupt();
- } catch (SecurityException ignore) {
- }
- }
- }
- }
-
// misc support for ForkJoinWorkerThread
/**
@@ -1152,17 +1128,21 @@ public class ForkJoinPool extends Abstra
}
/**
- * Accumulates steal count from a worker, clearing
- * the worker's value
+ * Tries to accumulates steal count from a worker, clearing
+ * the worker's value.
+ *
+ * @return true if worker steal count now zero
*/
- final void accumulateStealCount(ForkJoinWorkerThread w) {
+ final boolean tryAccumulateStealCount(ForkJoinWorkerThread w) {
int sc = w.stealCount;
- if (sc != 0) {
- long c;
- w.stealCount = 0;
- do {} while (!UNSAFE.compareAndSwapLong(this, stealCountOffset,
- c = stealCount, c + sc));
+ long c = stealCount;
+ // CAS even if zero, for fence effects
+ if (UNSAFE.compareAndSwapLong(this, stealCountOffset, c, c + sc)) {
+ if (sc != 0)
+ w.stealCount = 0;
+ return true;
}
+ return sc == 0;
}
/**
@@ -1245,7 +1225,7 @@ public class ForkJoinPool extends Abstra
checkPermission();
if (factory == null)
throw new NullPointerException();
- if (parallelism <= 0 || parallelism > MAX_THREADS)
+ if (parallelism <= 0 || parallelism > MAX_WORKERS)
throw new IllegalArgumentException();
this.parallelism = parallelism;
this.factory = factory;
@@ -1257,6 +1237,7 @@ public class ForkJoinPool extends Abstra
this.workerLock = new ReentrantLock();
this.termination = new Phaser(1);
this.poolNumber = poolNumberGenerator.incrementAndGet();
+ this.trimTime = System.nanoTime();
}
/**
@@ -1264,8 +1245,8 @@ public class ForkJoinPool extends Abstra
* @param pc the initial parallelism level
*/
private static int initialArraySizeFor(int pc) {
- // See Hackers Delight, sec 3.2. We know MAX_THREADS < (1 >>> 16)
- int size = pc < MAX_THREADS ? pc + 1 : MAX_THREADS;
+ // See Hackers Delight, sec 3.2. We know MAX_WORKERS < (1 >>> 16)
+ int size = pc < MAX_WORKERS ? pc + 1 : MAX_WORKERS;
size |= size >>> 1;
size |= size >>> 2;
size |= size >>> 4;
@@ -1284,8 +1265,8 @@ public class ForkJoinPool extends Abstra
if (runState >= SHUTDOWN)
throw new RejectedExecutionException();
submissionQueue.offer(task);
- signalEvent();
- ensureEnoughTotalWorkers();
+ advanceEventCount();
+ helpMaintainParallelism(); // start or wake up workers
}
/**
@@ -1532,8 +1513,8 @@ public class ForkJoinPool extends Abstra
public long getQueuedTaskCount() {
long count = 0;
ForkJoinWorkerThread[] ws = workers;
- int nws = ws.length;
- for (int i = 0; i < nws; ++i) {
+ int n = ws.length;
+ for (int i = 0; i < n; ++i) {
ForkJoinWorkerThread w = ws[i];
if (w != null)
count += w.getQueueSize();
@@ -1591,29 +1572,13 @@ public class ForkJoinPool extends Abstra
* @return the number of elements transferred
*/
protected int drainTasksTo(Collection> c) {
- int n = submissionQueue.drainTo(c);
- ForkJoinWorkerThread[] ws = workers;
- int nws = ws.length;
- for (int i = 0; i < nws; ++i) {
- ForkJoinWorkerThread w = ws[i];
- if (w != null)
- n += w.drainTasksTo(c);
- }
- return n;
- }
-
- /**
- * Returns count of total parks by existing workers.
- * Used during development only since not meaningful to users.
- */
- private int collectParkCount() {
- int count = 0;
+ int count = submissionQueue.drainTo(c);
ForkJoinWorkerThread[] ws = workers;
- int nws = ws.length;
- for (int i = 0; i < nws; ++i) {
+ int n = ws.length;
+ for (int i = 0; i < n; ++i) {
ForkJoinWorkerThread w = ws[i];
if (w != null)
- count += w.parkCount;
+ count += w.drainTasksTo(c);
}
return count;
}
@@ -1635,7 +1600,6 @@ public class ForkJoinPool extends Abstra
int pc = parallelism;
int rs = runState;
int ac = rs & ACTIVE_COUNT_MASK;
- // int pk = collectParkCount();
return super.toString() +
"[" + runLevelToString(rs) +
", parallelism = " + pc +
@@ -1645,7 +1609,6 @@ public class ForkJoinPool extends Abstra
", steals = " + st +
", tasks = " + qt +
", submissions = " + qs +
- // ", parks = " + pk +
"]";
}
@@ -1752,11 +1715,17 @@ public class ForkJoinPool extends Abstra
* Interface for extending managed parallelism for tasks running
* in {@link ForkJoinPool}s.
*
- * A {@code ManagedBlocker} provides two methods.
- * Method {@code isReleasable} must return {@code true} if
- * blocking is not necessary. Method {@code block} blocks the
- * current thread if necessary (perhaps internally invoking
- * {@code isReleasable} before actually blocking).
+ *
A {@code ManagedBlocker} provides two methods. Method
+ * {@code isReleasable} must return {@code true} if blocking is
+ * not necessary. Method {@code block} blocks the current thread
+ * if necessary (perhaps internally invoking {@code isReleasable}
+ * before actually blocking). The unusual methods in this API
+ * accommodate synchronizers that may, but don't usually, block
+ * for long periods. Similarly, they allow more efficient internal
+ * handling of cases in which additional workers may be, but
+ * usually are not, needed to ensure sufficient parallelism.
+ * Toward this end, implementations of method {@code isReleasable}
+ * must be amenable to repeated invocation.
*
*
For example, here is a ManagedBlocker based on a
* ReentrantLock:
@@ -1774,6 +1743,26 @@ public class ForkJoinPool extends Abstra
* return hasLock || (hasLock = lock.tryLock());
* }
* }}
+ *
+ *
Here is a class that possibly blocks waiting for an
+ * item on a given queue:
+ *
{@code
+ * class QueueTaker implements ManagedBlocker {
+ * final BlockingQueue queue;
+ * volatile E item = null;
+ * QueueTaker(BlockingQueue q) { this.queue = q; }
+ * public boolean block() throws InterruptedException {
+ * if (item == null)
+ * item = queue.take
+ * return true;
+ * }
+ * public boolean isReleasable() {
+ * return item != null || (item = queue.poll) != null;
+ * }
+ * public E getItem() { // call after pool.managedBlock completes
+ * return item;
+ * }
+ * }}
*/
public static interface ManagedBlocker {
/**
@@ -1816,8 +1805,10 @@ public class ForkJoinPool extends Abstra
public static void managedBlock(ManagedBlocker blocker)
throws InterruptedException {
Thread t = Thread.currentThread();
- if (t instanceof ForkJoinWorkerThread)
- ((ForkJoinWorkerThread) t).pool.awaitBlocker(blocker);
+ if (t instanceof ForkJoinWorkerThread) {
+ ForkJoinWorkerThread w = (ForkJoinWorkerThread) t;
+ w.pool.awaitBlocker(blocker);
+ }
else {
do {} while (!blocker.isReleasable() && !blocker.block());
}
@@ -1848,6 +1839,8 @@ public class ForkJoinPool extends Abstra
objectFieldOffset("eventWaiters",ForkJoinPool.class);
private static final long stealCountOffset =
objectFieldOffset("stealCount",ForkJoinPool.class);
+ private static final long spareWaitersOffset =
+ objectFieldOffset("spareWaiters",ForkJoinPool.class);
private static long objectFieldOffset(String field, Class klazz) {
try {