Renaissance

Benchmarking Suite for Parallel Applications on the JVM

Aleksandar Prokopec, Andrea Rosà, David Leopoldseder, Gilles Duboscq, Petr Tůma, Martin Studener, Lubomír Bulej, Yudi Zheng, Alex Villazón, Doug Simon, Thomas Würthinger, Walter Binder

Motivation

JVM Is Evolving

• New JVM features: concurrency, lambdas, method-handles, non-blocking I/O, ...
• New paradigms: data-parallelism, fork-join, streams, asynchronous programming with futures, transactional memory, ...
• New frameworks: Big Data (Spark), stream-processing (JDK8 Streams, Rx), actors (Akka), network communication (Netty) ...
• Common theme: concurrency and parallelism.
• Are there new kinds of applications to optimize?

Selection methodology

Goals

• Modern concurrency - cover many different styles.
• Realistic workloads - widely-used frameworks, production usages, existing benchmarks, mining.
• Workload diversity - stress different JVM features.
• Deterministic execution - to the extend possible.
• Open-source availability - licensing, debugging, source inspection.

Goals

• Modern concurrency - cover many different styles.
• Realistic workloads - widely-used frameworks, production usages, existing benchmarks, mining.
• Workload diversity - stress different JVM features.
• Deterministic execution - to the extend possible.
• Open-source availability - licensing, debugging, source inspection.

Gathered around 100 workloads, and used them to select 21 benchmarks.

Analysis

Metrics

PCA

Normalization over reference cycles.

Standardization to a [-1, 1] range.

Principal component analysis.

PCA: Components 1&2

PCA: Components 1&2

PCA: Components 3&4

PCA: Components 3&4

Atomic Operation Count

Invokedynamic Count

Synchronized Count

Are these differences relevant for performance?

Compiler optimizations

Escape Analysis with Atomics

Atomic reference objects, futures.

Escape Analysis with Atomics

Escape Analysis with Atomics

Loop-Wide Lock Coarsening

Synchronized collections.

Loop-Wide Lock Coarsening

Method-handle simplification

JDK8 Streams, Rx.

Method-handle simplification

MethodHandle reflection necessary.

Method-handle simplification

MethodHandle reflection necessary.

Method-handle simplification

Profile of the Scrabble benchmark.

Speculative Guard Motion

Speculative Guard Motion

Optimization Matrix

Graal vs C2 Differences

Code Complexity

Complexity Metrics

We computed the average and sum of all Chidamber & Kemerer metrics, and recorded minimum, maximum and average across benchmarks.

Conclusion: similar averages across suites (details in the paper).

Loaded Classes

Process

renaissance.dev

Open-Source Process

• Source code public, PRs welcome!
• Community can propose new benchmarks.
• Committee votes on benchmarks for the next release.
• Next steps: harness with detailed analyses for benchmark characterization, new compiler optimizations, effects on GC?

Open-Source Process

• Source code public, PRs welcome!
• Community can propose new benchmarks.
• Committee votes on benchmarks for the next release.
• Next steps: harness with detailed analyses for benchmark characterization, new compiler optimizations, effects on GC?

Goal: keep improving and evolving the suite in an open manner.

Thank you!