Posts Tagged ‘ProjectReactor’
[DevoxxPL2019] Constructing Custom Reactive Publishers: Insights into Project Reactor Internals
Lecturer
Oleh Dokuka contributes to Project Reactor as a committer, authoring books on reactive programming with Spring and serving as a software engineer at Superhuman. Based in Kyiv, he actively participates in conferences and communities focused on asynchronous systems.
Abstract
This inquiry delves into the intricacies of building a reactive publisher compliant with Reactive Streams specifications, drawing from Project Reactor’s design. It covers the rationale behind the spec, naive implementations, concurrency patterns like work-in-progress, and verification via TCK. Through iterative coding, it analyzes challenges in non-blocking data flows, backpressure, and thread safety, pondering effects on debugging, customization, and library extension.
Demystifying Reactive Streams: Specification and Purpose
Reactive Streams standardize asynchronous, non-blocking data processing with backpressure, addressing overflow in producer-consumer scenarios. Oleh commences by recalling the spec’s origins, crafted to unify libraries like RxJava and Akka Streams, ensuring interoperability.
Core interfaces—Publisher, Subscriber, Subscription, Processor—define interactions: publishers emit items, subscribers consume, subscriptions mediate requests and cancellations. The spec mandates rules for thread safety and signal ordering, preventing races.
Contextually, adoption surged with Java 9’s Flow API, embedding reactivity natively. Analytically, backpressure—subscribers requesting items—prevents buffering overloads, crucial in unbounded sources like networks.
Implications: enables composable, resilient pipelines, but demands adherence to 50+ rules, tested via TCK. For developers, understanding facilitates debugging; for extenders, it unlocks optimizations.
Naive Publisher Construction: Initial Steps and Pitfalls
Commencing with a basic array publisher, Oleh demonstrates emitting elements on subscription. Yet, naivety ignores concurrency: parallel subscriptions risk duplicates or misses.
Methodologically, extend TCK’s PublisherVerification for rule checks. Initial failures highlight needs for atomic operations and request tracking.
A subscription class manages emissions:
class ArraySubscription<T> implements Subscription {
private final Subscriber<? super T> subscriber;
private final T[] array;
private int index = 0;
private boolean canceled = false;
public ArraySubscription(Subscriber<? super T> subscriber, T[] array) {
this.subscriber = subscriber;
this.array = array;
}
@Override
public void request(long n) {
if (n <= 0 && !canceled) {
subscriber.onError(new IllegalArgumentException("Non-positive request"));
canceled = true;
return;
}
for (long i = 0; i < n && !canceled; i++) {
if (index < array.length) {
subscriber.onNext(array[index++]);
} else {
subscriber.onComplete();
canceled = true;
break;
}
}
}
@Override
public void cancel() {
canceled = true;
}
}
This handles basics but falters under concurrency, necessitating refinements.
Incorporating Concurrency Safeguards: Work-in-Progress and Atomicity
To thread-safely accumulate requests, introduce work-in-progress (WIP)—an atomic counter tracking processing state. Oleh explains: increment WIP to claim emission exclusivity; if non-zero, another thread processes, so defer.
Requests add to a requested counter atomically. On WIP decrement to zero, check if more requests pend, resuming if so.
This pattern, akin to semaphores, ensures single-threaded emission despite multi-threaded requests, averting races.
Analytically, it balances responsiveness and safety, though overflows (Long.MAX_VALUE) signal unbounded requests, potentially overwhelming subscribers.
Implications: facilitates non-blocking I/O, vital for high-throughput, but debugging requires tracing atomics.
Verification and Iterative Refinement: Ensuring Spec Compliance
Leverage TCK for exhaustive testing: extend PublisherVerification, supplying working and failing publishers. Tests validate signals, backpressure, and edge cases like negative requests.
Oleh iterates: failures prompt guards, like canceling on invalid requests. Post-fixes, all pass, confirming robustness.
Methodologically, TCK simulates parallelism, exposing flaws early. For custom operators, similar suites verify.
Consequences: empowers library creation or tweaks, as in optimizing for known guarantees, enhancing performance in specific flows.
Extending to Operators and Libraries: Building Beyond Basics
With a compliant publisher, assemble operators chaining transformations. Oleh hints at flux wrappers, where sources like arrays feed pipelines.
Analytically, operators preserve backpressure, propagating requests upstream. This composability yields expressive, efficient streams.
Implications: demystifies internals, aiding contributions to Reactor or custom variants for niches like low-latency trading.
In conclusion, mastering publishers unlocks reactive potential, transforming complex async into manageable flows.