ViewVC Help
View File | Revision Log | Show Annotations | Download File | Root Listing
root/jsr166/jsr166/src/jsr166y/Phaser.java
(Generate patch)

Comparing jsr166/src/jsr166y/Phaser.java (file contents):
Revision 1.47 by dl, Wed Jul 7 19:52:32 2010 UTC vs.
Revision 1.77 by jsr166, Mon Oct 17 23:37:19 2011 UTC

# Line 1 | Line 1
1   /*
2   * Written by Doug Lea with assistance from members of JCP JSR-166
3   * Expert Group and released to the public domain, as explained at
4 < * http://creativecommons.org/licenses/publicdomain
4 > * http://creativecommons.org/publicdomain/zero/1.0/
5   */
6  
7   package jsr166y;
8  
9 < import java.util.concurrent.*;
10 <
9 > import java.util.concurrent.TimeUnit;
10 > import java.util.concurrent.TimeoutException;
11   import java.util.concurrent.atomic.AtomicReference;
12   import java.util.concurrent.locks.LockSupport;
13  
# Line 34 | Line 34 | import java.util.concurrent.locks.LockSu
34   * Phaser} may be repeatedly awaited.  Method {@link
35   * #arriveAndAwaitAdvance} has effect analogous to {@link
36   * java.util.concurrent.CyclicBarrier#await CyclicBarrier.await}. Each
37 < * generation of a {@code Phaser} has an associated phase number. The
38 < * phase number starts at zero, and advances when all parties arrive
39 < * at the barrier, wrapping around to zero after reaching {@code
37 > * generation of a phaser has an associated phase number. The phase
38 > * number starts at zero, and advances when all parties arrive at the
39 > * phaser, wrapping around to zero after reaching {@code
40   * Integer.MAX_VALUE}. The use of phase numbers enables independent
41 < * control of actions upon arrival at a barrier and upon awaiting
41 > * control of actions upon arrival at a phaser and upon awaiting
42   * others, via two kinds of methods that may be invoked by any
43   * registered party:
44   *
45   * <ul>
46   *
47   *   <li> <b>Arrival.</b> Methods {@link #arrive} and
48 < *       {@link #arriveAndDeregister} record arrival at a
49 < *       barrier. These methods do not block, but return an associated
50 < *       <em>arrival phase number</em>; that is, the phase number of
51 < *       the barrier to which the arrival applied. When the final
52 < *       party for a given phase arrives, an optional barrier action
53 < *       is performed and the phase advances.  Barrier actions,
54 < *       performed by the party triggering a phase advance, are
55 < *       arranged by overriding method {@link #onAdvance(int, int)},
56 < *       which also controls termination. Overriding this method is
57 < *       similar to, but more flexible than, providing a barrier
58 < *       action to a {@code CyclicBarrier}.
48 > *       {@link #arriveAndDeregister} record arrival.  These methods
49 > *       do not block, but return an associated <em>arrival phase
50 > *       number</em>; that is, the phase number of the phaser to which
51 > *       the arrival applied. When the final party for a given phase
52 > *       arrives, an optional action is performed and the phase
53 > *       advances.  These actions are performed by the party
54 > *       triggering a phase advance, and are arranged by overriding
55 > *       method {@link #onAdvance(int, int)}, which also controls
56 > *       termination. Overriding this method is similar to, but more
57 > *       flexible than, providing a barrier action to a {@code
58 > *       CyclicBarrier}.
59   *
60   *   <li> <b>Waiting.</b> Method {@link #awaitAdvance} requires an
61   *       argument indicating an arrival phase number, and returns when
62 < *       the barrier advances to (or is already at) a different phase.
62 > *       the phaser advances to (or is already at) a different phase.
63   *       Unlike similar constructions using {@code CyclicBarrier},
64   *       method {@code awaitAdvance} continues to wait even if the
65   *       waiting thread is interrupted. Interruptible and timeout
66   *       versions are also available, but exceptions encountered while
67   *       tasks wait interruptibly or with timeout do not change the
68 < *       state of the barrier. If necessary, you can perform any
68 > *       state of the phaser. If necessary, you can perform any
69   *       associated recovery within handlers of those exceptions,
70   *       often after invoking {@code forceTermination}.  Phasers may
71   *       also be used by tasks executing in a {@link ForkJoinPool},
# Line 74 | Line 74 | import java.util.concurrent.locks.LockSu
74   *
75   * </ul>
76   *
77 < * <p> <b>Termination.</b> A {@code Phaser} may enter a
78 < * <em>termination</em> state in which all synchronization methods
79 < * immediately return without updating phaser state or waiting for
80 < * advance, and indicating (via a negative phase value) that execution
81 < * is complete.  Termination is triggered when an invocation of {@code
82 < * onAdvance} returns {@code true}.  As illustrated below, when
83 < * phasers control actions with a fixed number of iterations, it is
84 < * often convenient to override this method to cause termination when
85 < * the current phase number reaches a threshold. Method {@link
86 < * #forceTermination} is also available to abruptly release waiting
87 < * threads and allow them to terminate.
88 < *
89 < * <p> <b>Tiering.</b> Phasers may be <em>tiered</em> (i.e., arranged
90 < * in tree structures) to reduce contention. Phasers with large
91 < * numbers of parties that would otherwise experience heavy
77 > * <p> <b>Termination.</b> A phaser may enter a <em>termination</em>
78 > * state, that may be checked using method {@link #isTerminated}. Upon
79 > * termination, all synchronization methods immediately return without
80 > * waiting for advance, as indicated by a negative return value.
81 > * Similarly, attempts to register upon termination have no effect.
82 > * Termination is triggered when an invocation of {@code onAdvance}
83 > * returns {@code true}. The default implementation returns {@code
84 > * true} if a deregistration has caused the number of registered
85 > * parties to become zero.  As illustrated below, when phasers control
86 > * actions with a fixed number of iterations, it is often convenient
87 > * to override this method to cause termination when the current phase
88 > * number reaches a threshold. Method {@link #forceTermination} is
89 > * also available to abruptly release waiting threads and allow them
90 > * to terminate.
91 > *
92 > * <p> <b>Tiering.</b> Phasers may be <em>tiered</em> (i.e.,
93 > * constructed in tree structures) to reduce contention. Phasers with
94 > * large numbers of parties that would otherwise experience heavy
95   * synchronization contention costs may instead be set up so that
96   * groups of sub-phasers share a common parent.  This may greatly
97   * increase throughput even though it incurs greater per-operation
98   * overhead.
99   *
100 + * <p>In a tree of tiered phasers, registration and deregistration of
101 + * child phasers with their parent are managed automatically.
102 + * Whenever the number of registered parties of a child phaser becomes
103 + * non-zero (as established in the {@link #Phaser(Phaser,int)}
104 + * constructor, {@link #register}, or {@link #bulkRegister}), the
105 + * child phaser is registered with its parent.  Whenever the number of
106 + * registered parties becomes zero as the result of an invocation of
107 + * {@link #arriveAndDeregister}, the child phaser is deregistered
108 + * from its parent.
109 + *
110   * <p><b>Monitoring.</b> While synchronization methods may be invoked
111   * only by registered parties, the current state of a phaser may be
112   * monitored by any caller.  At any given moment there are {@link
# Line 109 | Line 122 | import java.util.concurrent.locks.LockSu
122   * <p><b>Sample usages:</b>
123   *
124   * <p>A {@code Phaser} may be used instead of a {@code CountDownLatch}
125 < * to control a one-shot action serving a variable number of
126 < * parties. The typical idiom is for the method setting this up to
127 < * first register, then start the actions, then deregister, as in:
125 > * to control a one-shot action serving a variable number of parties.
126 > * The typical idiom is for the method setting this up to first
127 > * register, then start the actions, then deregister, as in:
128   *
129   *  <pre> {@code
130   * void runTasks(List<Runnable> tasks) {
131   *   final Phaser phaser = new Phaser(1); // "1" to register self
132   *   // create and start threads
133 < *   for (Runnable task : tasks) {
133 > *   for (final Runnable task : tasks) {
134   *     phaser.register();
135   *     new Thread() {
136   *       public void run() {
# Line 181 | Line 194 | import java.util.concurrent.locks.LockSu
194   * }}</pre>
195   *
196   *
197 < * <p>To create a set of tasks using a tree of phasers,
198 < * you could use code of the following form, assuming a
199 < * Task class with a constructor accepting a phaser that
200 < * it registers for upon construction:
197 > * <p>To create a set of {@code n} tasks using a tree of phasers, you
198 > * could use code of the following form, assuming a Task class with a
199 > * constructor accepting a {@code Phaser} that it registers with upon
200 > * construction. After invocation of {@code build(new Task[n], 0, n,
201 > * new Phaser())}, these tasks could then be started, for example by
202 > * submitting to a pool:
203   *
204   *  <pre> {@code
205 < * void build(Task[] actions, int lo, int hi, Phaser ph) {
205 > * void build(Task[] tasks, int lo, int hi, Phaser ph) {
206   *   if (hi - lo > TASKS_PER_PHASER) {
207   *     for (int i = lo; i < hi; i += TASKS_PER_PHASER) {
208   *       int j = Math.min(i + TASKS_PER_PHASER, hi);
209 < *       build(actions, i, j, new Phaser(ph));
209 > *       build(tasks, i, j, new Phaser(ph));
210   *     }
211   *   } else {
212   *     for (int i = lo; i < hi; ++i)
213 < *       actions[i] = new Task(ph);
213 > *       tasks[i] = new Task(ph);
214   *       // assumes new Task(ph) performs ph.register()
215   *   }
216 < * }
202 < * // .. initially called, for n tasks via
203 < * build(new Task[n], 0, n, new Phaser());}</pre>
216 > * }}</pre>
217   *
218   * The best value of {@code TASKS_PER_PHASER} depends mainly on
219 < * expected barrier synchronization rates. A value as low as four may
220 < * be appropriate for extremely small per-barrier task bodies (thus
219 > * expected synchronization rates. A value as low as four may
220 > * be appropriate for extremely small per-phase task bodies (thus
221   * high rates), or up to hundreds for extremely large ones.
222   *
210 * </pre>
211 *
223   * <p><b>Implementation notes</b>: This implementation restricts the
224   * maximum number of parties to 65535. Attempts to register additional
225   * parties result in {@code IllegalStateException}. However, you can and
# Line 226 | Line 237 | public class Phaser {
237       */
238  
239      /**
240 <     * Barrier state representation. Conceptually, a barrier contains
230 <     * four values:
231 <     *
232 <     * * parties -- the number of parties to wait (16 bits)
233 <     * * unarrived -- the number of parties yet to hit barrier (16 bits)
234 <     * * phase -- the generation of the barrier (31 bits)
235 <     * * terminated -- set if barrier is terminated (1 bit)
236 <     *
237 <     * However, to efficiently maintain atomicity, these values are
238 <     * packed into a single (atomic) long. Termination uses the sign
239 <     * bit of 32 bit representation of phase, so phase is set to -1 on
240 <     * termination. Good performance relies on keeping state decoding
241 <     * and encoding simple, and keeping race windows short.
240 >     * Primary state representation, holding four bit-fields:
241       *
242 <     * Note: there are some cheats in arrive() that rely on unarrived
243 <     * count being lowest 16 bits.
242 >     * unarrived  -- the number of parties yet to hit barrier (bits  0-15)
243 >     * parties    -- the number of parties to wait            (bits 16-31)
244 >     * phase      -- the generation of the barrier            (bits 32-62)
245 >     * terminated -- set if barrier is terminated             (bit  63 / sign)
246 >     *
247 >     * Except that a phaser with no registered parties is
248 >     * distinguished by the otherwise illegal state of having zero
249 >     * parties and one unarrived parties (encoded as EMPTY below).
250 >     *
251 >     * To efficiently maintain atomicity, these values are packed into
252 >     * a single (atomic) long. Good performance relies on keeping
253 >     * state decoding and encoding simple, and keeping race windows
254 >     * short.
255 >     *
256 >     * All state updates are performed via CAS except initial
257 >     * registration of a sub-phaser (i.e., one with a non-null
258 >     * parent).  In this (relatively rare) case, we use built-in
259 >     * synchronization to lock while first registering with its
260 >     * parent.
261 >     *
262 >     * The phase of a subphaser is allowed to lag that of its
263 >     * ancestors until it is actually accessed -- see method
264 >     * reconcileState.
265       */
266      private volatile long state;
267  
268 <    private static final int ushortMask = 0xffff;
269 <    private static final int phaseMask  = 0x7fffffff;
268 >    private static final int  MAX_PARTIES     = 0xffff;
269 >    private static final int  MAX_PHASE       = Integer.MAX_VALUE;
270 >    private static final int  PARTIES_SHIFT   = 16;
271 >    private static final int  PHASE_SHIFT     = 32;
272 >    private static final int  UNARRIVED_MASK  = 0xffff;      // to mask ints
273 >    private static final long PARTIES_MASK    = 0xffff0000L; // to mask longs
274 >    private static final long COUNTS_MASK     = 0xffffffffL;
275 >    private static final long TERMINATION_BIT = 1L << 63;
276 >
277 >    // some special values
278 >    private static final int  ONE_ARRIVAL     = 1;
279 >    private static final int  ONE_PARTY       = 1 << PARTIES_SHIFT;
280 >    private static final int  ONE_DEREGISTER  = ONE_ARRIVAL|ONE_PARTY;
281 >    private static final int  EMPTY           = 1;
282 >
283 >    // The following unpacking methods are usually manually inlined
284  
285      private static int unarrivedOf(long s) {
286 <        return (int) (s & ushortMask);
286 >        int counts = (int)s;
287 >        return (counts == EMPTY) ? 0 : (counts & UNARRIVED_MASK);
288      }
289  
290      private static int partiesOf(long s) {
291 <        return ((int) s) >>> 16;
291 >        return (int)s >>> PARTIES_SHIFT;
292      }
293  
294      private static int phaseOf(long s) {
295 <        return (int) (s >>> 32);
295 >        return (int)(s >>> PHASE_SHIFT);
296      }
297  
298      private static int arrivedOf(long s) {
299 <        return partiesOf(s) - unarrivedOf(s);
300 <    }
301 <
267 <    private static long stateFor(int phase, int parties, int unarrived) {
268 <        return ((((long) phase) << 32) | (((long) parties) << 16) |
269 <                (long) unarrived);
270 <    }
271 <
272 <    private static long trippedStateFor(int phase, int parties) {
273 <        long lp = (long) parties;
274 <        return (((long) phase) << 32) | (lp << 16) | lp;
275 <    }
276 <
277 <    /**
278 <     * Returns message string for bad bounds exceptions.
279 <     */
280 <    private static String badBounds(int parties, int unarrived) {
281 <        return ("Attempt to set " + unarrived +
282 <                " unarrived of " + parties + " parties");
299 >        int counts = (int)s;
300 >        return (counts == EMPTY) ? 0 :
301 >            (counts >>> PARTIES_SHIFT) - (counts & UNARRIVED_MASK);
302      }
303  
304      /**
# Line 288 | Line 307 | public class Phaser {
307      private final Phaser parent;
308  
309      /**
310 <     * The root of phaser tree. Equals this if not in a tree.  Used to
292 <     * support faster state push-down.
310 >     * The root of phaser tree. Equals this if not in a tree.
311       */
312      private final Phaser root;
313  
296    // Wait queues
297
314      /**
315       * Heads of Treiber stacks for waiting threads. To eliminate
316 <     * contention while releasing some threads while adding others, we
316 >     * contention when releasing some threads while adding others, we
317       * use two of them, alternating across even and odd phases.
318 +     * Subphasers share queues with root to speed up releases.
319       */
320 <    private final AtomicReference<QNode> evenQ = new AtomicReference<QNode>();
321 <    private final AtomicReference<QNode> oddQ  = new AtomicReference<QNode>();
320 >    private final AtomicReference<QNode> evenQ;
321 >    private final AtomicReference<QNode> oddQ;
322  
323      private AtomicReference<QNode> queueFor(int phase) {
324          return ((phase & 1) == 0) ? evenQ : oddQ;
325      }
326  
327      /**
328 <     * Returns current state, first resolving lagged propagation from
312 <     * root if necessary.
328 >     * Returns message string for bounds exceptions on arrival.
329       */
330 <    private long getReconciledState() {
331 <        return (parent == null) ? state : reconcileState();
330 >    private String badArrive(long s) {
331 >        return "Attempted arrival of unregistered party for " +
332 >            stateToString(s);
333      }
334  
335      /**
336 <     * Recursively resolves state.
336 >     * Returns message string for bounds exceptions on registration.
337       */
338 <    private long reconcileState() {
339 <        Phaser p = parent;
340 <        long s = state;
341 <        if (p != null) {
342 <            while (unarrivedOf(s) == 0 && phaseOf(s) != phaseOf(root.state)) {
343 <                long parentState = p.getReconciledState();
344 <                int parentPhase = phaseOf(parentState);
345 <                int phase = phaseOf(s = state);
346 <                if (phase != parentPhase) {
347 <                    long next = trippedStateFor(parentPhase, partiesOf(s));
348 <                    if (casState(s, next)) {
338 >    private String badRegister(long s) {
339 >        return "Attempt to register more than " +
340 >            MAX_PARTIES + " parties for " + stateToString(s);
341 >    }
342 >
343 >    /**
344 >     * Main implementation for methods arrive and arriveAndDeregister.
345 >     * Manually tuned to speed up and minimize race windows for the
346 >     * common case of just decrementing unarrived field.
347 >     *
348 >     * @param adjust value to subtract from state;
349 >     *               ONE_ARRIVAL for arrive,
350 >     *               ONE_DEREGISTER for arriveAndDeregister
351 >     */
352 >    private int doArrive(int adjust) {
353 >        final Phaser root = this.root;
354 >        for (;;) {
355 >            long s = (root == this) ? state : reconcileState();
356 >            int phase = (int)(s >>> PHASE_SHIFT);
357 >            if (phase < 0)
358 >                return phase;
359 >            int counts = (int)s;
360 >            int unarrived = (counts == EMPTY) ? 0 : (counts & UNARRIVED_MASK);
361 >            if (unarrived <= 0)
362 >                throw new IllegalStateException(badArrive(s));
363 >            if (UNSAFE.compareAndSwapLong(this, stateOffset, s, s-=adjust)) {
364 >                if (unarrived == 1) {
365 >                    long n = s & PARTIES_MASK;  // base of next state
366 >                    int nextUnarrived = (int)n >>> PARTIES_SHIFT;
367 >                    if (root == this) {
368 >                        if (onAdvance(phase, nextUnarrived))
369 >                            n |= TERMINATION_BIT;
370 >                        else if (nextUnarrived == 0)
371 >                            n |= EMPTY;
372 >                        else
373 >                            n |= nextUnarrived;
374 >                        int nextPhase = (phase + 1) & MAX_PHASE;
375 >                        n |= (long)nextPhase << PHASE_SHIFT;
376 >                        UNSAFE.compareAndSwapLong(this, stateOffset, s, n);
377                          releaseWaiters(phase);
378 <                        s = next;
378 >                    }
379 >                    else if (nextUnarrived == 0) { // propagate deregistration
380 >                        phase = parent.doArrive(ONE_DEREGISTER);
381 >                        UNSAFE.compareAndSwapLong(this, stateOffset,
382 >                                                  s, s | EMPTY);
383 >                    }
384 >                    else
385 >                        phase = parent.doArrive(ONE_ARRIVAL);
386 >                }
387 >                return phase;
388 >            }
389 >        }
390 >    }
391 >
392 >    /**
393 >     * Implementation of register, bulkRegister
394 >     *
395 >     * @param registrations number to add to both parties and
396 >     * unarrived fields. Must be greater than zero.
397 >     */
398 >    private int doRegister(int registrations) {
399 >        // adjustment to state
400 >        long adjust = ((long)registrations << PARTIES_SHIFT) | registrations;
401 >        final Phaser parent = this.parent;
402 >        int phase;
403 >        for (;;) {
404 >            long s = (parent == null) ? state : reconcileState();
405 >            int counts = (int)s;
406 >            int parties = counts >>> PARTIES_SHIFT;
407 >            int unarrived = counts & UNARRIVED_MASK;
408 >            if (registrations > MAX_PARTIES - parties)
409 >                throw new IllegalStateException(badRegister(s));
410 >            phase = (int)(s >>> PHASE_SHIFT);
411 >            if (phase < 0)
412 >                break;
413 >            if (counts != EMPTY) {                  // not 1st registration
414 >                if (parent == null || reconcileState() == s) {
415 >                    if (unarrived == 0)             // wait out advance
416 >                        root.internalAwaitAdvance(phase, null);
417 >                    else if (UNSAFE.compareAndSwapLong(this, stateOffset,
418 >                                                       s, s + adjust))
419 >                        break;
420 >                }
421 >            }
422 >            else if (parent == null) {              // 1st root registration
423 >                long next = ((long)phase << PHASE_SHIFT) | adjust;
424 >                if (UNSAFE.compareAndSwapLong(this, stateOffset, s, next))
425 >                    break;
426 >            }
427 >            else {
428 >                synchronized (this) {               // 1st sub registration
429 >                    if (state == s) {               // recheck under lock
430 >                        phase = parent.doRegister(1);
431 >                        if (phase < 0)
432 >                            break;
433 >                        // finish registration whenever parent registration
434 >                        // succeeded, even when racing with termination,
435 >                        // since these are part of the same "transaction".
436 >                        while (!UNSAFE.compareAndSwapLong
437 >                               (this, stateOffset, s,
438 >                                ((long)phase << PHASE_SHIFT) | adjust)) {
439 >                            s = state;
440 >                            phase = (int)(root.state >>> PHASE_SHIFT);
441 >                            // assert (int)s == EMPTY;
442 >                        }
443 >                        break;
444                      }
445                  }
446              }
447          }
448 +        return phase;
449 +    }
450 +
451 +    /**
452 +     * Resolves lagged phase propagation from root if necessary.
453 +     * Reconciliation normally occurs when root has advanced but
454 +     * subphasers have not yet done so, in which case they must finish
455 +     * their own advance by setting unarrived to parties (or if
456 +     * parties is zero, resetting to unregistered EMPTY state).
457 +     *
458 +     * @return reconciled state
459 +     */
460 +    private long reconcileState() {
461 +        final Phaser root = this.root;
462 +        long s = state;
463 +        if (root != this) {
464 +            int phase, p;
465 +            // CAS to root phase with current parties, tripping unarrived
466 +            while ((phase = (int)(root.state >>> PHASE_SHIFT)) !=
467 +                   (int)(s >>> PHASE_SHIFT) &&
468 +                   !UNSAFE.compareAndSwapLong
469 +                   (this, stateOffset, s,
470 +                    s = (((long)phase << PHASE_SHIFT) |
471 +                         ((phase < 0) ? (s & COUNTS_MASK) :
472 +                          (((p = (int)s >>> PARTIES_SHIFT) == 0) ? EMPTY :
473 +                           ((s & PARTIES_MASK) | p))))))
474 +                s = state;
475 +        }
476          return s;
477      }
478  
479      /**
480 <     * Creates a new phaser without any initially registered parties,
481 <     * initial phase number 0, and no parent. Any thread using this
480 >     * Creates a new phaser with no initially registered parties, no
481 >     * parent, and initial phase number 0. Any thread using this
482       * phaser will need to first register for it.
483       */
484      public Phaser() {
485 <        this(null);
485 >        this(null, 0);
486      }
487  
488      /**
489 <     * Creates a new phaser with the given numbers of registered
490 <     * unarrived parties, initial phase number 0, and no parent.
489 >     * Creates a new phaser with the given number of registered
490 >     * unarrived parties, no parent, and initial phase number 0.
491       *
492 <     * @param parties the number of parties required to trip barrier
492 >     * @param parties the number of parties required to advance to the
493 >     * next phase
494       * @throws IllegalArgumentException if parties less than zero
495       * or greater than the maximum number of parties supported
496       */
# Line 360 | Line 499 | public class Phaser {
499      }
500  
501      /**
502 <     * Creates a new phaser with the given parent, without any
364 <     * initially registered parties. If parent is non-null this phaser
365 <     * is registered with the parent and its initial phase number is
366 <     * the same as that of parent phaser.
502 >     * Equivalent to {@link #Phaser(Phaser, int) Phaser(parent, 0)}.
503       *
504       * @param parent the parent phaser
505       */
506      public Phaser(Phaser parent) {
507 <        int phase = 0;
372 <        this.parent = parent;
373 <        if (parent != null) {
374 <            this.root = parent.root;
375 <            phase = parent.register();
376 <        }
377 <        else
378 <            this.root = this;
379 <        this.state = trippedStateFor(phase, 0);
507 >        this(parent, 0);
508      }
509  
510      /**
511 <     * Creates a new phaser with the given parent and numbers of
512 <     * registered unarrived parties. If parent is non-null, this phaser
513 <     * is registered with the parent and its initial phase number is
514 <     * the same as that of parent phaser.
511 >     * Creates a new phaser with the given parent and number of
512 >     * registered unarrived parties.  When the given parent is non-null
513 >     * and the given number of parties is greater than zero, this
514 >     * child phaser is registered with its parent.
515       *
516       * @param parent the parent phaser
517 <     * @param parties the number of parties required to trip barrier
517 >     * @param parties the number of parties required to advance to the
518 >     * next phase
519       * @throws IllegalArgumentException if parties less than zero
520       * or greater than the maximum number of parties supported
521       */
522      public Phaser(Phaser parent, int parties) {
523 <        if (parties < 0 || parties > ushortMask)
523 >        if (parties >>> PARTIES_SHIFT != 0)
524              throw new IllegalArgumentException("Illegal number of parties");
525          int phase = 0;
526          this.parent = parent;
527          if (parent != null) {
528 <            this.root = parent.root;
529 <            phase = parent.register();
528 >            final Phaser root = parent.root;
529 >            this.root = root;
530 >            this.evenQ = root.evenQ;
531 >            this.oddQ = root.oddQ;
532 >            if (parties != 0)
533 >                phase = parent.doRegister(1);
534          }
535 <        else
535 >        else {
536              this.root = this;
537 <        this.state = trippedStateFor(phase, parties);
537 >            this.evenQ = new AtomicReference<QNode>();
538 >            this.oddQ = new AtomicReference<QNode>();
539 >        }
540 >        this.state = (parties == 0) ? (long)EMPTY :
541 >            ((long)phase << PHASE_SHIFT) |
542 >            ((long)parties << PARTIES_SHIFT) |
543 >            ((long)parties);
544      }
545  
546      /**
547 <     * Adds a new unarrived party to this phaser.
548 <     *
549 <     * @return the arrival phase number to which this registration applied
547 >     * Adds a new unarrived party to this phaser.  If an ongoing
548 >     * invocation of {@link #onAdvance} is in progress, this method
549 >     * may await its completion before returning.  If this phaser has
550 >     * a parent, and this phaser previously had no registered parties,
551 >     * this child phaser is also registered with its parent. If
552 >     * this phaser is terminated, the attempt to register has
553 >     * no effect, and a negative value is returned.
554 >     *
555 >     * @return the arrival phase number to which this registration
556 >     * applied.  If this value is negative, then this phaser has
557 >     * terminated, in which case registration has no effect.
558       * @throws IllegalStateException if attempting to register more
559       * than the maximum supported number of parties
560       */
# Line 417 | Line 564 | public class Phaser {
564  
565      /**
566       * Adds the given number of new unarrived parties to this phaser.
567 <     *
568 <     * @param parties the number of parties required to trip barrier
569 <     * @return the arrival phase number to which this registration applied
567 >     * If an ongoing invocation of {@link #onAdvance} is in progress,
568 >     * this method may await its completion before returning.  If this
569 >     * phaser has a parent, and the given number of parties is greater
570 >     * than zero, and this phaser previously had no registered
571 >     * parties, this child phaser is also registered with its parent.
572 >     * If this phaser is terminated, the attempt to register has no
573 >     * effect, and a negative value is returned.
574 >     *
575 >     * @param parties the number of additional parties required to
576 >     * advance to the next phase
577 >     * @return the arrival phase number to which this registration
578 >     * applied.  If this value is negative, then this phaser has
579 >     * terminated, in which case registration has no effect.
580       * @throws IllegalStateException if attempting to register more
581       * than the maximum supported number of parties
582 +     * @throws IllegalArgumentException if {@code parties < 0}
583       */
584      public int bulkRegister(int parties) {
585          if (parties < 0)
# Line 432 | Line 590 | public class Phaser {
590      }
591  
592      /**
593 <     * Shared code for register, bulkRegister
594 <     */
595 <    private int doRegister(int registrations) {
596 <        int phase;
597 <        for (;;) {
598 <            long s = getReconciledState();
441 <            phase = phaseOf(s);
442 <            int unarrived = unarrivedOf(s) + registrations;
443 <            int parties = partiesOf(s) + registrations;
444 <            if (phase < 0)
445 <                break;
446 <            if (parties > ushortMask || unarrived > ushortMask)
447 <                throw new IllegalStateException(badBounds(parties, unarrived));
448 <            if (phase == phaseOf(root.state) &&
449 <                casState(s, stateFor(phase, parties, unarrived)))
450 <                break;
451 <        }
452 <        return phase;
453 <    }
454 <
455 <    /**
456 <     * Arrives at the barrier, but does not wait for others.  (You can
457 <     * in turn wait for others via {@link #awaitAdvance}).  It is an
458 <     * unenforced usage error for an unregistered party to invoke this
459 <     * method.
593 >     * Arrives at this phaser, without waiting for others to arrive.
594 >     *
595 >     * <p>It is a usage error for an unregistered party to invoke this
596 >     * method.  However, this error may result in an {@code
597 >     * IllegalStateException} only upon some subsequent operation on
598 >     * this phaser, if ever.
599       *
600       * @return the arrival phase number, or a negative value if terminated
601       * @throws IllegalStateException if not terminated and the number
602       * of unarrived parties would become negative
603       */
604      public int arrive() {
605 <        int phase;
467 <        for (;;) {
468 <            long s = state;
469 <            phase = phaseOf(s);
470 <            if (phase < 0)
471 <                break;
472 <            int parties = partiesOf(s);
473 <            int unarrived = unarrivedOf(s) - 1;
474 <            if (unarrived > 0) {        // Not the last arrival
475 <                if (casState(s, s - 1)) // s-1 adds one arrival
476 <                    break;
477 <            }
478 <            else if (unarrived == 0) {  // the last arrival
479 <                Phaser par = parent;
480 <                if (par == null) {      // directly trip
481 <                    if (casState
482 <                        (s,
483 <                         trippedStateFor(onAdvance(phase, parties) ? -1 :
484 <                                         ((phase + 1) & phaseMask), parties))) {
485 <                        releaseWaiters(phase);
486 <                        break;
487 <                    }
488 <                }
489 <                else {                  // cascade to parent
490 <                    if (casState(s, s - 1)) { // zeroes unarrived
491 <                        par.arrive();
492 <                        reconcileState();
493 <                        break;
494 <                    }
495 <                }
496 <            }
497 <            else if (phase != phaseOf(root.state)) // or if unreconciled
498 <                reconcileState();
499 <            else
500 <                throw new IllegalStateException(badBounds(parties, unarrived));
501 <        }
502 <        return phase;
605 >        return doArrive(ONE_ARRIVAL);
606      }
607  
608      /**
609 <     * Arrives at the barrier and deregisters from it without waiting
610 <     * for others. Deregistration reduces the number of parties
611 <     * required to trip the barrier in future phases.  If this phaser
609 >     * Arrives at this phaser and deregisters from it without waiting
610 >     * for others to arrive. Deregistration reduces the number of
611 >     * parties required to advance in future phases.  If this phaser
612       * has a parent, and deregistration causes this phaser to have
613 <     * zero parties, this phaser also arrives at and is deregistered
614 <     * from its parent.  It is an unenforced usage error for an
615 <     * unregistered party to invoke this method.
613 >     * zero parties, this phaser is also deregistered from its parent.
614 >     *
615 >     * <p>It is a usage error for an unregistered party to invoke this
616 >     * method.  However, this error may result in an {@code
617 >     * IllegalStateException} only upon some subsequent operation on
618 >     * this phaser, if ever.
619       *
620       * @return the arrival phase number, or a negative value if terminated
621       * @throws IllegalStateException if not terminated and the number
622       * of registered or unarrived parties would become negative
623       */
624      public int arriveAndDeregister() {
625 <        // similar code to arrive, but too different to merge
520 <        Phaser par = parent;
521 <        int phase;
522 <        for (;;) {
523 <            long s = state;
524 <            phase = phaseOf(s);
525 <            if (phase < 0)
526 <                break;
527 <            int parties = partiesOf(s) - 1;
528 <            int unarrived = unarrivedOf(s) - 1;
529 <            if (parties >= 0) {
530 <                if (unarrived > 0 || (unarrived == 0 && par != null)) {
531 <                    if (casState
532 <                        (s,
533 <                         stateFor(phase, parties, unarrived))) {
534 <                        if (unarrived == 0) {
535 <                            par.arriveAndDeregister();
536 <                            reconcileState();
537 <                        }
538 <                        break;
539 <                    }
540 <                    continue;
541 <                }
542 <                if (unarrived == 0) {
543 <                    if (casState
544 <                        (s,
545 <                         trippedStateFor(onAdvance(phase, parties) ? -1 :
546 <                                         ((phase + 1) & phaseMask), parties))) {
547 <                        releaseWaiters(phase);
548 <                        break;
549 <                    }
550 <                    continue;
551 <                }
552 <                if (par != null && phase != phaseOf(root.state)) {
553 <                    reconcileState();
554 <                    continue;
555 <                }
556 <            }
557 <            throw new IllegalStateException(badBounds(parties, unarrived));
558 <        }
559 <        return phase;
625 >        return doArrive(ONE_DEREGISTER);
626      }
627  
628      /**
629 <     * Arrives at the barrier and awaits others. Equivalent in effect
629 >     * Arrives at this phaser and awaits others. Equivalent in effect
630       * to {@code awaitAdvance(arrive())}.  If you need to await with
631       * interruption or timeout, you can arrange this with an analogous
632 <     * construction using one of the other forms of the awaitAdvance
633 <     * method.  If instead you need to deregister upon arrival use
634 <     * {@code arriveAndDeregister}. It is an unenforced usage error
635 <     * for an unregistered party to invoke this method.
632 >     * construction using one of the other forms of the {@code
633 >     * awaitAdvance} method.  If instead you need to deregister upon
634 >     * arrival, use {@code awaitAdvance(arriveAndDeregister())}.
635 >     *
636 >     * <p>It is a usage error for an unregistered party to invoke this
637 >     * method.  However, this error may result in an {@code
638 >     * IllegalStateException} only upon some subsequent operation on
639 >     * this phaser, if ever.
640       *
641 <     * @return the arrival phase number, or a negative number if terminated
641 >     * @return the arrival phase number, or the (negative)
642 >     * {@linkplain #getPhase() current phase} if terminated
643       * @throws IllegalStateException if not terminated and the number
644       * of unarrived parties would become negative
645       */
646      public int arriveAndAwaitAdvance() {
647 <        return awaitAdvance(arrive());
647 >        // Specialization of doArrive+awaitAdvance eliminating some reads/paths
648 >        final Phaser root = this.root;
649 >        for (;;) {
650 >            long s = (root == this) ? state : reconcileState();
651 >            int phase = (int)(s >>> PHASE_SHIFT);
652 >            if (phase < 0)
653 >                return phase;
654 >            int counts = (int)s;
655 >            int unarrived = (counts == EMPTY) ? 0 : (counts & UNARRIVED_MASK);
656 >            if (unarrived <= 0)
657 >                throw new IllegalStateException(badArrive(s));
658 >            if (UNSAFE.compareAndSwapLong(this, stateOffset, s,
659 >                                          s -= ONE_ARRIVAL)) {
660 >                if (unarrived > 1)
661 >                    return root.internalAwaitAdvance(phase, null);
662 >                if (root != this)
663 >                    return parent.arriveAndAwaitAdvance();
664 >                long n = s & PARTIES_MASK;  // base of next state
665 >                int nextUnarrived = (int)n >>> PARTIES_SHIFT;
666 >                if (onAdvance(phase, nextUnarrived))
667 >                    n |= TERMINATION_BIT;
668 >                else if (nextUnarrived == 0)
669 >                    n |= EMPTY;
670 >                else
671 >                    n |= nextUnarrived;
672 >                int nextPhase = (phase + 1) & MAX_PHASE;
673 >                n |= (long)nextPhase << PHASE_SHIFT;
674 >                if (!UNSAFE.compareAndSwapLong(this, stateOffset, s, n))
675 >                    return (int)(state >>> PHASE_SHIFT); // terminated
676 >                releaseWaiters(phase);
677 >                return nextPhase;
678 >            }
679 >        }
680      }
681  
682      /**
683 <     * Awaits the phase of the barrier to advance from the given phase
684 <     * value, returning immediately if the current phase of the
685 <     * barrier is not equal to the given phase value or this barrier
583 <     * is terminated.  It is an unenforced usage error for an
584 <     * unregistered party to invoke this method.
683 >     * Awaits the phase of this phaser to advance from the given phase
684 >     * value, returning immediately if the current phase is not equal
685 >     * to the given phase value or this phaser is terminated.
686       *
687       * @param phase an arrival phase number, or negative value if
688       * terminated; this argument is normally the value returned by a
689 <     * previous call to {@code arrive} or its variants
690 <     * @return the next arrival phase number, or a negative value
691 <     * if terminated or argument is negative
689 >     * previous call to {@code arrive} or {@code arriveAndDeregister}.
690 >     * @return the next arrival phase number, or the argument if it is
691 >     * negative, or the (negative) {@linkplain #getPhase() current phase}
692 >     * if terminated
693       */
694      public int awaitAdvance(int phase) {
695 +        final Phaser root = this.root;
696 +        long s = (root == this) ? state : reconcileState();
697 +        int p = (int)(s >>> PHASE_SHIFT);
698          if (phase < 0)
699              return phase;
700 <        long s = getReconciledState();
701 <        int p = phaseOf(s);
702 <        if (p != phase)
598 <            return p;
599 <        if (unarrivedOf(s) == 0 && parent != null)
600 <            parent.awaitAdvance(phase);
601 <        // Fall here even if parent waited, to reconcile and help release
602 <        return untimedWait(phase);
700 >        if (p == phase)
701 >            return root.internalAwaitAdvance(phase, null);
702 >        return p;
703      }
704  
705      /**
706 <     * Awaits the phase of the barrier to advance from the given phase
706 >     * Awaits the phase of this phaser to advance from the given phase
707       * value, throwing {@code InterruptedException} if interrupted
708 <     * while waiting, or returning immediately if the current phase of
709 <     * the barrier is not equal to the given phase value or this
710 <     * barrier is terminated. It is an unenforced usage error for an
611 <     * unregistered party to invoke this method.
708 >     * while waiting, or returning immediately if the current phase is
709 >     * not equal to the given phase value or this phaser is
710 >     * terminated.
711       *
712       * @param phase an arrival phase number, or negative value if
713       * terminated; this argument is normally the value returned by a
714 <     * previous call to {@code arrive} or its variants
715 <     * @return the next arrival phase number, or a negative value
716 <     * if terminated or argument is negative
714 >     * previous call to {@code arrive} or {@code arriveAndDeregister}.
715 >     * @return the next arrival phase number, or the argument if it is
716 >     * negative, or the (negative) {@linkplain #getPhase() current phase}
717 >     * if terminated
718       * @throws InterruptedException if thread interrupted while waiting
719       */
720      public int awaitAdvanceInterruptibly(int phase)
721          throws InterruptedException {
722 +        final Phaser root = this.root;
723 +        long s = (root == this) ? state : reconcileState();
724 +        int p = (int)(s >>> PHASE_SHIFT);
725          if (phase < 0)
726              return phase;
727 <        long s = getReconciledState();
728 <        int p = phaseOf(s);
729 <        if (p != phase)
730 <            return p;
731 <        if (unarrivedOf(s) == 0 && parent != null)
732 <            parent.awaitAdvanceInterruptibly(phase);
733 <        return interruptibleWait(phase);
727 >        if (p == phase) {
728 >            QNode node = new QNode(this, phase, true, false, 0L);
729 >            p = root.internalAwaitAdvance(phase, node);
730 >            if (node.wasInterrupted)
731 >                throw new InterruptedException();
732 >        }
733 >        return p;
734      }
735  
736      /**
737 <     * Awaits the phase of the barrier to advance from the given phase
737 >     * Awaits the phase of this phaser to advance from the given phase
738       * value or the given timeout to elapse, throwing {@code
739       * InterruptedException} if interrupted while waiting, or
740 <     * returning immediately if the current phase of the barrier is
741 <     * not equal to the given phase value or this barrier is
639 <     * terminated.  It is an unenforced usage error for an
640 <     * unregistered party to invoke this method.
740 >     * returning immediately if the current phase is not equal to the
741 >     * given phase value or this phaser is terminated.
742       *
743       * @param phase an arrival phase number, or negative value if
744       * terminated; this argument is normally the value returned by a
745 <     * previous call to {@code arrive} or its variants
745 >     * previous call to {@code arrive} or {@code arriveAndDeregister}.
746       * @param timeout how long to wait before giving up, in units of
747       *        {@code unit}
748       * @param unit a {@code TimeUnit} determining how to interpret the
749       *        {@code timeout} parameter
750 <     * @return the next arrival phase number, or a negative value
751 <     * if terminated or argument is negative
750 >     * @return the next arrival phase number, or the argument if it is
751 >     * negative, or the (negative) {@linkplain #getPhase() current phase}
752 >     * if terminated
753       * @throws InterruptedException if thread interrupted while waiting
754       * @throws TimeoutException if timed out while waiting
755       */
756      public int awaitAdvanceInterruptibly(int phase,
757                                           long timeout, TimeUnit unit)
758          throws InterruptedException, TimeoutException {
759 +        long nanos = unit.toNanos(timeout);
760 +        final Phaser root = this.root;
761 +        long s = (root == this) ? state : reconcileState();
762 +        int p = (int)(s >>> PHASE_SHIFT);
763          if (phase < 0)
764              return phase;
765 <        long s = getReconciledState();
766 <        int p = phaseOf(s);
767 <        if (p != phase)
768 <            return p;
769 <        if (unarrivedOf(s) == 0 && parent != null)
770 <            parent.awaitAdvanceInterruptibly(phase, timeout, unit);
771 <        return timedWait(phase, unit.toNanos(timeout));
765 >        if (p == phase) {
766 >            QNode node = new QNode(this, phase, true, true, nanos);
767 >            p = root.internalAwaitAdvance(phase, node);
768 >            if (node.wasInterrupted)
769 >                throw new InterruptedException();
770 >            else if (p == phase)
771 >                throw new TimeoutException();
772 >        }
773 >        return p;
774      }
775  
776      /**
777 <     * Forces this barrier to enter termination state. Counts of
778 <     * arrived and registered parties are unaffected. If this phaser
779 <     * has a parent, it too is terminated. This method may be useful
780 <     * for coordinating recovery after one or more tasks encounter
777 >     * Forces this phaser to enter termination state.  Counts of
778 >     * registered parties are unaffected.  If this phaser is a member
779 >     * of a tiered set of phasers, then all of the phasers in the set
780 >     * are terminated.  If this phaser is already terminated, this
781 >     * method has no effect.  This method may be useful for
782 >     * coordinating recovery after one or more tasks encounter
783       * unexpected exceptions.
784       */
785      public void forceTermination() {
786 <        for (;;) {
787 <            long s = getReconciledState();
788 <            int phase = phaseOf(s);
789 <            int parties = partiesOf(s);
790 <            int unarrived = unarrivedOf(s);
791 <            if (phase < 0 ||
792 <                casState(s, stateFor(-1, parties, unarrived))) {
793 <                releaseWaiters(0);
794 <                releaseWaiters(1);
685 <                if (parent != null)
686 <                    parent.forceTermination();
786 >        // Only need to change root state
787 >        final Phaser root = this.root;
788 >        long s;
789 >        while ((s = root.state) >= 0) {
790 >            if (UNSAFE.compareAndSwapLong(root, stateOffset,
791 >                                          s, s | TERMINATION_BIT)) {
792 >                // signal all threads
793 >                releaseWaiters(0); // Waiters on evenQ
794 >                releaseWaiters(1); // Waiters on oddQ
795                  return;
796              }
797          }
# Line 692 | Line 800 | public class Phaser {
800      /**
801       * Returns the current phase number. The maximum phase number is
802       * {@code Integer.MAX_VALUE}, after which it restarts at
803 <     * zero. Upon termination, the phase number is negative.
803 >     * zero. Upon termination, the phase number is negative,
804 >     * in which case the prevailing phase prior to termination
805 >     * may be obtained via {@code getPhase() + Integer.MIN_VALUE}.
806       *
807       * @return the phase number, or a negative value if terminated
808       */
809      public final int getPhase() {
810 <        return phaseOf(getReconciledState());
810 >        return (int)(root.state >>> PHASE_SHIFT);
811      }
812  
813      /**
814 <     * Returns the number of parties registered at this barrier.
814 >     * Returns the number of parties registered at this phaser.
815       *
816       * @return the number of parties
817       */
# Line 711 | Line 821 | public class Phaser {
821  
822      /**
823       * Returns the number of registered parties that have arrived at
824 <     * the current phase of this barrier.
824 >     * the current phase of this phaser. If this phaser has terminated,
825 >     * the returned value is meaningless and arbitrary.
826       *
827       * @return the number of arrived parties
828       */
829      public int getArrivedParties() {
830 <        return arrivedOf(state);
830 >        return arrivedOf(reconcileState());
831      }
832  
833      /**
834       * Returns the number of registered parties that have not yet
835 <     * arrived at the current phase of this barrier.
835 >     * arrived at the current phase of this phaser. If this phaser has
836 >     * terminated, the returned value is meaningless and arbitrary.
837       *
838       * @return the number of unarrived parties
839       */
840      public int getUnarrivedParties() {
841 <        return unarrivedOf(state);
841 >        return unarrivedOf(reconcileState());
842      }
843  
844      /**
# Line 749 | Line 861 | public class Phaser {
861      }
862  
863      /**
864 <     * Returns {@code true} if this barrier has been terminated.
864 >     * Returns {@code true} if this phaser has been terminated.
865       *
866 <     * @return {@code true} if this barrier has been terminated
866 >     * @return {@code true} if this phaser has been terminated
867       */
868      public boolean isTerminated() {
869 <        return getPhase() < 0;
869 >        return root.state < 0L;
870      }
871  
872      /**
873       * Overridable method to perform an action upon impending phase
874       * advance, and to control termination. This method is invoked
875 <     * upon arrival of the party tripping the barrier (when all other
875 >     * upon arrival of the party advancing this phaser (when all other
876       * waiting parties are dormant).  If this method returns {@code
877 <     * true}, then, rather than advance the phase number, this barrier
878 <     * will be set to a final termination state, and subsequent calls
879 <     * to {@link #isTerminated} will return true. Any (unchecked)
880 <     * Exception or Error thrown by an invocation of this method is
881 <     * propagated to the party attempting to trip the barrier, in
882 <     * which case no advance occurs.
877 >     * true}, this phaser will be set to a final termination state
878 >     * upon advance, and subsequent calls to {@link #isTerminated}
879 >     * will return true. Any (unchecked) Exception or Error thrown by
880 >     * an invocation of this method is propagated to the party
881 >     * attempting to advance this phaser, in which case no advance
882 >     * occurs.
883       *
884       * <p>The arguments to this method provide the state of the phaser
885 <     * prevailing for the current transition. (When called from within
886 <     * an implementation of {@code onAdvance} the values returned by
887 <     * methods such as {@code getPhase} may or may not reliably
888 <     * indicate the state to which this transition applies.)
889 <     *
890 <     * <p>The default version returns {@code true} when the number of
891 <     * registered parties is zero. Normally, overrides that arrange
892 <     * termination for other reasons should also preserve this
893 <     * property.
894 <     *
895 <     * <p>You may override this method to perform an action with side
896 <     * effects visible to participating tasks, but it is only sensible
897 <     * to do so in designs where all parties register before any
898 <     * arrive, and all {@link #awaitAdvance} at each phase.
899 <     * Otherwise, you cannot ensure lack of interference from other
900 <     * parties during the invocation of this method. Additionally,
901 <     * method {@code onAdvance} may be invoked more than once per
902 <     * transition if registrations are intermixed with arrivals.
885 >     * prevailing for the current transition.  The effects of invoking
886 >     * arrival, registration, and waiting methods on this phaser from
887 >     * within {@code onAdvance} are unspecified and should not be
888 >     * relied on.
889 >     *
890 >     * <p>If this phaser is a member of a tiered set of phasers, then
891 >     * {@code onAdvance} is invoked only for its root phaser on each
892 >     * advance.
893 >     *
894 >     * <p>To support the most common use cases, the default
895 >     * implementation of this method returns {@code true} when the
896 >     * number of registered parties has become zero as the result of a
897 >     * party invoking {@code arriveAndDeregister}.  You can disable
898 >     * this behavior, thus enabling continuation upon future
899 >     * registrations, by overriding this method to always return
900 >     * {@code false}:
901 >     *
902 >     * <pre> {@code
903 >     * Phaser phaser = new Phaser() {
904 >     *   protected boolean onAdvance(int phase, int parties) { return false; }
905 >     * }}</pre>
906       *
907 <     * @param phase the phase number on entering the barrier
907 >     * @param phase the current phase number on entry to this method,
908 >     * before this phaser is advanced
909       * @param registeredParties the current number of registered parties
910 <     * @return {@code true} if this barrier should terminate
910 >     * @return {@code true} if this phaser should terminate
911       */
912      protected boolean onAdvance(int phase, int registeredParties) {
913 <        return registeredParties <= 0;
913 >        return registeredParties == 0;
914      }
915  
916      /**
# Line 804 | Line 920 | public class Phaser {
920       * followed by the number of registered parties, and {@code
921       * "arrived = "} followed by the number of arrived parties.
922       *
923 <     * @return a string identifying this barrier, as well as its state
923 >     * @return a string identifying this phaser, as well as its state
924       */
925      public String toString() {
926 <        long s = getReconciledState();
926 >        return stateToString(reconcileState());
927 >    }
928 >
929 >    /**
930 >     * Implementation of toString and string-based error messages
931 >     */
932 >    private String stateToString(long s) {
933          return super.toString() +
934              "[phase = " + phaseOf(s) +
935              " parties = " + partiesOf(s) +
936              " arrived = " + arrivedOf(s) + "]";
937      }
938  
939 <    // methods for waiting
939 >    // Waiting mechanics
940  
941      /**
942 <     * Wait nodes for Treiber stack representing wait queue
942 >     * Removes and signals threads from queue for phase.
943       */
944 <    static final class QNode implements ForkJoinPool.ManagedBlocker {
945 <        final Phaser phaser;
946 <        final int phase;
947 <        final long startTime;
948 <        final long nanos;
949 <        final boolean timed;
950 <        final boolean interruptible;
951 <        volatile boolean wasInterrupted = false;
952 <        volatile Thread thread; // nulled to cancel wait
831 <        QNode next;
832 <        QNode(Phaser phaser, int phase, boolean interruptible,
833 <              boolean timed, long startTime, long nanos) {
834 <            this.phaser = phaser;
835 <            this.phase = phase;
836 <            this.timed = timed;
837 <            this.interruptible = interruptible;
838 <            this.startTime = startTime;
839 <            this.nanos = nanos;
840 <            thread = Thread.currentThread();
841 <        }
842 <        public boolean isReleasable() {
843 <            return (thread == null ||
844 <                    phaser.getPhase() != phase ||
845 <                    (interruptible && wasInterrupted) ||
846 <                    (timed && (nanos - (System.nanoTime() - startTime)) <= 0));
847 <        }
848 <        public boolean block() {
849 <            if (Thread.interrupted()) {
850 <                wasInterrupted = true;
851 <                if (interruptible)
852 <                    return true;
853 <            }
854 <            if (!timed)
855 <                LockSupport.park(this);
856 <            else {
857 <                long waitTime = nanos - (System.nanoTime() - startTime);
858 <                if (waitTime <= 0)
859 <                    return true;
860 <                LockSupport.parkNanos(this, waitTime);
861 <            }
862 <            return isReleasable();
863 <        }
864 <        void signal() {
865 <            Thread t = thread;
866 <            if (t != null) {
867 <                thread = null;
944 >    private void releaseWaiters(int phase) {
945 >        QNode q;   // first element of queue
946 >        Thread t;  // its thread
947 >        AtomicReference<QNode> head = (phase & 1) == 0 ? evenQ : oddQ;
948 >        while ((q = head.get()) != null &&
949 >               q.phase != (int)(root.state >>> PHASE_SHIFT)) {
950 >            if (head.compareAndSet(q, q.next) &&
951 >                (t = q.thread) != null) {
952 >                q.thread = null;
953                  LockSupport.unpark(t);
954              }
955          }
871        boolean doWait() {
872            if (thread != null) {
873                try {
874                    ForkJoinPool.managedBlock(this);
875                } catch (InterruptedException ie) {
876                }
877            }
878            return wasInterrupted;
879        }
880
956      }
957  
958      /**
959 <     * Removes and signals waiting threads from wait queue.
959 >     * Variant of releaseWaiters that additionally tries to remove any
960 >     * nodes no longer waiting for advance due to timeout or
961 >     * interrupt. Currently, nodes are removed only if they are at
962 >     * head of queue, which suffices to reduce memory footprint in
963 >     * most usages.
964 >     *
965 >     * @return current phase on exit
966       */
967 <    private void releaseWaiters(int phase) {
968 <        AtomicReference<QNode> head = queueFor(phase);
969 <        QNode q;
970 <        while ((q = head.get()) != null) {
971 <            if (head.compareAndSet(q, q.next))
972 <                q.signal();
967 >    private int abortWait(int phase) {
968 >        AtomicReference<QNode> head = (phase & 1) == 0 ? evenQ : oddQ;
969 >        for (;;) {
970 >            Thread t;
971 >            QNode q = head.get();
972 >            int p = (int)(root.state >>> PHASE_SHIFT);
973 >            if (q == null || ((t = q.thread) != null && q.phase == p))
974 >                return p;
975 >            if (head.compareAndSet(q, q.next) && t != null) {
976 >                q.thread = null;
977 >                LockSupport.unpark(t);
978 >            }
979          }
980      }
981  
982 +    /** The number of CPUs, for spin control */
983 +    private static final int NCPU = Runtime.getRuntime().availableProcessors();
984 +
985      /**
986 <     * Tries to enqueue given node in the appropriate wait queue.
987 <     *
988 <     * @return true if successful
986 >     * The number of times to spin before blocking while waiting for
987 >     * advance, per arrival while waiting. On multiprocessors, fully
988 >     * blocking and waking up a large number of threads all at once is
989 >     * usually a very slow process, so we use rechargeable spins to
990 >     * avoid it when threads regularly arrive: When a thread in
991 >     * internalAwaitAdvance notices another arrival before blocking,
992 >     * and there appear to be enough CPUs available, it spins
993 >     * SPINS_PER_ARRIVAL more times before blocking. The value trades
994 >     * off good-citizenship vs big unnecessary slowdowns.
995       */
996 <    private boolean tryEnqueue(QNode node) {
901 <        AtomicReference<QNode> head = queueFor(node.phase);
902 <        return head.compareAndSet(node.next = head.get(), node);
903 <    }
996 >    static final int SPINS_PER_ARRIVAL = (NCPU < 2) ? 1 : 1 << 8;
997  
998      /**
999 <     * Enqueues node and waits unless aborted or signalled.
999 >     * Possibly blocks and waits for phase to advance unless aborted.
1000 >     * Call only on root phaser.
1001       *
1002 +     * @param phase current phase
1003 +     * @param node if non-null, the wait node to track interrupt and timeout;
1004 +     * if null, denotes noninterruptible wait
1005       * @return current phase
1006       */
1007 <    private int untimedWait(int phase) {
1008 <        QNode node = null;
1009 <        boolean queued = false;
1010 <        boolean interrupted = false;
1007 >    private int internalAwaitAdvance(int phase, QNode node) {
1008 >        // assert root == this;
1009 >        releaseWaiters(phase-1);          // ensure old queue clean
1010 >        boolean queued = false;           // true when node is enqueued
1011 >        int lastUnarrived = 0;            // to increase spins upon change
1012 >        int spins = SPINS_PER_ARRIVAL;
1013 >        long s;
1014          int p;
1015 <        while ((p = getPhase()) == phase) {
1016 <            if (Thread.interrupted())
1017 <                interrupted = true;
1018 <            else if (node == null)
1019 <                node = new QNode(this, phase, false, false, 0, 0);
1020 <            else if (!queued)
1021 <                queued = tryEnqueue(node);
1022 <            else
1023 <                interrupted = node.doWait();
1015 >        while ((p = (int)((s = state) >>> PHASE_SHIFT)) == phase) {
1016 >            if (node == null) {           // spinning in noninterruptible mode
1017 >                int unarrived = (int)s & UNARRIVED_MASK;
1018 >                if (unarrived != lastUnarrived &&
1019 >                    (lastUnarrived = unarrived) < NCPU)
1020 >                    spins += SPINS_PER_ARRIVAL;
1021 >                boolean interrupted = Thread.interrupted();
1022 >                if (interrupted || --spins < 0) { // need node to record intr
1023 >                    node = new QNode(this, phase, false, false, 0L);
1024 >                    node.wasInterrupted = interrupted;
1025 >                }
1026 >            }
1027 >            else if (node.isReleasable()) // done or aborted
1028 >                break;
1029 >            else if (!queued) {           // push onto queue
1030 >                AtomicReference<QNode> head = (phase & 1) == 0 ? evenQ : oddQ;
1031 >                QNode q = node.next = head.get();
1032 >                if ((q == null || q.phase == phase) &&
1033 >                    (int)(state >>> PHASE_SHIFT) == phase) // avoid stale enq
1034 >                    queued = head.compareAndSet(q, node);
1035 >            }
1036 >            else {
1037 >                try {
1038 >                    ForkJoinPool.managedBlock(node);
1039 >                } catch (InterruptedException ie) {
1040 >                    node.wasInterrupted = true;
1041 >                }
1042 >            }
1043 >        }
1044 >
1045 >        if (node != null) {
1046 >            if (node.thread != null)
1047 >                node.thread = null;       // avoid need for unpark()
1048 >            if (node.wasInterrupted && !node.interruptible)
1049 >                Thread.currentThread().interrupt();
1050 >            if (p == phase && (p = (int)(state >>> PHASE_SHIFT)) == phase)
1051 >                return abortWait(phase); // possibly clean up on abort
1052          }
925        if (node != null)
926            node.thread = null;
1053          releaseWaiters(phase);
928        if (interrupted)
929            Thread.currentThread().interrupt();
1054          return p;
1055      }
1056  
1057      /**
1058 <     * Interruptible version
935 <     * @return current phase
1058 >     * Wait nodes for Treiber stack representing wait queue
1059       */
1060 <    private int interruptibleWait(int phase) throws InterruptedException {
1061 <        QNode node = null;
1062 <        boolean queued = false;
1063 <        boolean interrupted = false;
1064 <        int p;
1065 <        while ((p = getPhase()) == phase && !interrupted) {
1066 <            if (Thread.interrupted())
1067 <                interrupted = true;
1068 <            else if (node == null)
1069 <                node = new QNode(this, phase, true, false, 0, 0);
947 <            else if (!queued)
948 <                queued = tryEnqueue(node);
949 <            else
950 <                interrupted = node.doWait();
951 <        }
952 <        if (node != null)
953 <            node.thread = null;
954 <        if (p != phase || (p = getPhase()) != phase)
955 <            releaseWaiters(phase);
956 <        if (interrupted)
957 <            throw new InterruptedException();
958 <        return p;
959 <    }
1060 >    static final class QNode implements ForkJoinPool.ManagedBlocker {
1061 >        final Phaser phaser;
1062 >        final int phase;
1063 >        final boolean interruptible;
1064 >        final boolean timed;
1065 >        boolean wasInterrupted;
1066 >        long nanos;
1067 >        long lastTime;
1068 >        volatile Thread thread; // nulled to cancel wait
1069 >        QNode next;
1070  
1071 <    /**
1072 <     * Timeout version.
1073 <     * @return current phase
1074 <     */
1075 <    private int timedWait(int phase, long nanos)
1076 <        throws InterruptedException, TimeoutException {
1077 <        long startTime = System.nanoTime();
1078 <        QNode node = null;
1079 <        boolean queued = false;
1080 <        boolean interrupted = false;
1081 <        int p;
1082 <        while ((p = getPhase()) == phase && !interrupted) {
1071 >        QNode(Phaser phaser, int phase, boolean interruptible,
1072 >              boolean timed, long nanos) {
1073 >            this.phaser = phaser;
1074 >            this.phase = phase;
1075 >            this.interruptible = interruptible;
1076 >            this.nanos = nanos;
1077 >            this.timed = timed;
1078 >            this.lastTime = timed ? System.nanoTime() : 0L;
1079 >            thread = Thread.currentThread();
1080 >        }
1081 >
1082 >        public boolean isReleasable() {
1083 >            if (thread == null)
1084 >                return true;
1085 >            if (phaser.getPhase() != phase) {
1086 >                thread = null;
1087 >                return true;
1088 >            }
1089              if (Thread.interrupted())
1090 <                interrupted = true;
1091 <            else if (nanos - (System.nanoTime() - startTime) <= 0)
1092 <                break;
1093 <            else if (node == null)
1094 <                node = new QNode(this, phase, true, true, startTime, nanos);
1095 <            else if (!queued)
1096 <                queued = tryEnqueue(node);
1097 <            else
1098 <                interrupted = node.doWait();
1099 <        }
1100 <        if (node != null)
1101 <            node.thread = null;
1102 <        if (p != phase || (p = getPhase()) != phase)
1103 <            releaseWaiters(phase);
1104 <        if (interrupted)
1105 <            throw new InterruptedException();
1106 <        if (p == phase)
1107 <            throw new TimeoutException();
1108 <        return p;
1090 >                wasInterrupted = true;
1091 >            if (wasInterrupted && interruptible) {
1092 >                thread = null;
1093 >                return true;
1094 >            }
1095 >            if (timed) {
1096 >                if (nanos > 0L) {
1097 >                    long now = System.nanoTime();
1098 >                    nanos -= now - lastTime;
1099 >                    lastTime = now;
1100 >                }
1101 >                if (nanos <= 0L) {
1102 >                    thread = null;
1103 >                    return true;
1104 >                }
1105 >            }
1106 >            return false;
1107 >        }
1108 >
1109 >        public boolean block() {
1110 >            if (isReleasable())
1111 >                return true;
1112 >            else if (!timed)
1113 >                LockSupport.park(this);
1114 >            else if (nanos > 0)
1115 >                LockSupport.parkNanos(this, nanos);
1116 >            return isReleasable();
1117 >        }
1118      }
1119  
1120      // Unsafe mechanics
1121  
1122 <    private static final sun.misc.Unsafe UNSAFE = getUnsafe();
1123 <    private static final long stateOffset =
1124 <        objectFieldOffset("state", Phaser.class);
1000 <
1001 <    private final boolean casState(long cmp, long val) {
1002 <        return UNSAFE.compareAndSwapLong(this, stateOffset, cmp, val);
1003 <    }
1004 <
1005 <    private static long objectFieldOffset(String field, Class<?> klazz) {
1122 >    private static final sun.misc.Unsafe UNSAFE;
1123 >    private static final long stateOffset;
1124 >    static {
1125          try {
1126 <            return UNSAFE.objectFieldOffset(klazz.getDeclaredField(field));
1127 <        } catch (NoSuchFieldException e) {
1128 <            // Convert Exception to corresponding Error
1129 <            NoSuchFieldError error = new NoSuchFieldError(field);
1130 <            error.initCause(e);
1131 <            throw error;
1126 >            UNSAFE = getUnsafe();
1127 >            Class<?> k = Phaser.class;
1128 >            stateOffset = UNSAFE.objectFieldOffset
1129 >                (k.getDeclaredField("state"));
1130 >        } catch (Exception e) {
1131 >            throw new Error(e);
1132          }
1133      }
1134  

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines