Evaluating the Jazz Transformer: Surprise, Risk, and Human Practice

2026-11-10

The Jazz Transformer (Wu & Yang, ISMIR 2020) is a Transformer-XL trained to generate jazz solos as sequences of events: bars, positions, chords, pitches, durations. Training and evaluation are largely based on fit to the dataset distribution, with rigid definitions of "correctness". There are no metrics implemented to evaluate the creativity of generated compositions, so we leverage information theory and the interplay of human surprise and uncertainty to quantify risk. Ultimately, we are curious about whether a generated solo takes risks the way a human improviser does.

This research asks a narrow question: when a model takes a risk, how justified is that risk in the context of jazz?

The short version: at sampling temperature T=1.2T = 1.2, generated solos take risks at a slightly lower rate than human players, but continue after them and fit them to the chords in a human way. Raising the temperature changes how many risks the model takes much more than how it handles them. A chord-blind Markov baseline also looks human by JJ, but chord fit separates it from both humans and the transformer, and only the transformer's harmonic gain disappears when it is given the wrong chords.


Setup

All results use the pretrained checkpoint jazz-trsfmr-B-loss0.25: a 12-layer Transformer-XL (dmodel=512d_{\text{model}} = 512, 8 heads, 41M parameters) over a vocabulary of 469 event ids, with 64 positions per 4/4 bar.

ItemDetail
Human corpusWeimar Jazz Database (WJazzD), 431 solos in 4/4; 409 used to train the judges, 22 held out
Generated set10 solos of 32 bars at temperature 1.2
Markov baselineTwo first-order chains with no chords, fit on 372 WJazzD training solos; 22 solos of 32 bars; see the baseline section below

Everything measured by the "judges" below is in bits per note.


Judging surprise with Markov models

To talk about risk-taking, we need an outside model of what's expected in a jazz line. We used smoothed Markov "judges" from the PPM/IDyOM family, which predict each note's melodic interval (pitch minus previous pitch, in semitones). They're trained on the 409 WJazzD training songs: 178,018 notes and 74 distinct intervals.

Contexts c0,…,cmc_0, \dots, c_m run from most to least specific. Starting from a uniform floor over V+1V + 1 outcomes (the extra slot covers unseen intervals), each level seen in training is mixed in:

Pfloor(w)=1V+1,Pk(w)=count(ck,w)+T(ck) Pk+1(w)N(ck)+T(ck)P_{\text{floor}}(w) = \frac{1}{V+1}, \qquad P_k(w) = \frac{\mathrm{count}(c_k, w) + T(c_k)\,P_{k+1}(w)}{N(c_k) + T(c_k)}

where T(c)T(c) is the number of distinct intervals seen after context cc and N(c)N(c) is its total count. For each note, the information content is IC=−log⁡2P(actual interval)IC = -\log_2 P(\text{actual interval}) and HH is the entropy of the predicted distribution.

There are four judges:

JudgeContextRole
A, melody-onlyPrevious 2 intervals, backing offExpectation from melodic habit
H, harmony-only(previous note's chord degree, chord quality, beat)Expectation from harmony and meter
B, chord-awareGeometric mix of A and HThe main risk judge
S, short-termA's chain at order 3, counts reset for each soloPredictability from the solo's own material

Judge B combines the melody and harmony judges the way IDyOM combines viewpoints:

PB(x)∝PA(x)1−w PH(x)wP_B(x) \propto P_A(x)^{1-w} \, P_H(x)^{w}

The weight ww is fitted by maximum held-out likelihood over 5 cross-validation folds grouped by artist, so no artist is in both the fitting and the scoring set. The likelihood curve has a clean interior minimum at w=0.38w = 0.38, and every mix between 0.1 and 0.7 beats both pure models.

JudgeReal solos, IC (bits)Generated solos, IC (bits)
A, melody-only3.363.44
H, harmony-only3.503.43
B, chord-aware3.293.31
S, short-term4.074.40

Two things stand out. Generated solos are about as typical as real ones note to note (A: 3.44 vs 3.36). But they're 0.33 bits less predictable from their own material (S), which means the model reuses its own motifs less than human players do. Harmony also matters more for the generated solos: harmony-only H actually beats melody-only A on them, but not on real solos, so the model's lines follow the chords more closely than real players' lines do.


Risk, and whether it's handled like a human

A note is surprising when it carries more information than the judge expected:

risk=ICB−HB\text{risk} = IC_B - H_B

A note is a risk event when its risk exceeds τrisk\tau_{\text{risk}}, the 95th percentile of risk on cross-validated human data (3.13 bits).

Surprise alone doesn't say whether a risk worked. In jazz, an outside note is justified by what comes next: a line that steps into a chord tone makes the risk sound intentional. So we built a retrospective judge RR that also sees the next two notes. A future-aware model FF predicts the same interval from the previous interval, the span to the next note, the next note's chord degree, and the span after that. RR combines FF with BB the same way BB combines AA and HH:

PR(x)∝PB(x)1−v PF(x)v,J=ICB−ICRP_R(x) \propto P_B(x)^{1-v} \, P_F(x)^{v}, \qquad J = IC_B - IC_R

JJ is the number of bits of the note's surprise that its continuation explains: the pointwise mutual information between the note and what follows, given the past. At v=0v = 0, R=BR = B and J=0J = 0. Fitting vv the same way gives v=0.52v = 0.52. JJ is positive when the line steps to a chord tone and negative before a leap.

Alongside JJ, chord fit scores how well the risk note and its next two notes sit on the harmony:

fitwin=∑i=02[log⁡2P(degreei∣chord typei)+log⁡212]\text{fit}_{\text{win}} = \sum_{i=0}^{2} \left[ \log_2 P(\text{degree}_i \mid \text{chord type}_i) + \log_2 12 \right]

That's the number of bits each note's degree gains over a harmony-blind, uniform pitch class. Over a dominant seventh, for example, the root scores +0.77 bits and the major 7th scores −0.97. On human risks, JJ and chord fit are uncorrelated (r=0.01r = 0.01), so they're kept as separate measures.

Neither score needs labels. Both are calibrated on held-out human risk events (8,860 of them), and a risk is flagged when it falls below the human 5th percentile. By construction, 5% of human risks get flagged, which makes "human practice" the reference point. A set of solos is then compared with humans by AUC, where 0.5 means indistinguishable.

So each set of solos gets three numbers: how many risks it takes (surprise), how it continues after them (JJ), and how well they fit the chords (chord fit).

Results

At T=1.2T = 1.2 the transformer is close to human practice on all three:

MeasureHuman (22 held-out WJazzD solos)Transformer, T=1.2T = 1.2 (10)Markov, WJazzD fit (22)
Risk events per 100 notes4.43.86.5
JJ: flagged at 5% false-alarm rate4.7%1.9%2.9%
JJ: AUC against human risks0.510.520.48
Chord fit: flagged3.5%2.9%11.3%
Chord fit: AUC against human risks0.480.460.67
Chord fit: mean over risk windows (bits)+0.36+0.42−0.47

The transformer takes slightly fewer risks than humans, continues after them like a human, and fits the chords at least as well. With only 105 generated risk events, none of these gaps is meaningful. The Markov column is a chord-blind baseline, covered in the baseline section below: JJ can't tell it from human, but chord fit can.

Rationalizing the judge models

We validated the measures by damaging held-out human solos in ways with known answers and checking whether the scores catch the damage. Each corrupted set is compared with human risks of the same surprise, so surprise alone scores AUC 0.50 by design and only what JJ or chord fit adds can separate them.

CorruptionJJ AUCChord-fit AUC
Wrong note (non-chord tone not resolved by step)0.820.71
Continuation swapped in from another solo, ignoring the risk note0.840.66
Continuation swapped in, moving on from the risk note but ignoring the harmony0.440.64
Planted resolution (control, should look human)0.520.55

JJ catches wrong notes and lines that don't follow from the risk, and it does better than a simple "is it followed by a step?" rule. It misses smooth continuations that ignore the chords, because the interval span nearly pins down the risk note's pitch either way. Chord fit covers exactly that blind spot. Neither penalizes a well-formed resolution.

A temperature sweep tells the same story. Raising TT raises the risk rate (1.2 per 100 notes at T=0.8T = 0.8, 7.7 at T=2.0T = 2.0) and mean surprise; both rank perfectly with temperature (Spearman 1.0). JJ stays human-like throughout: even at T=2.0T = 2.0 the model follows 61% of its risks with a step, against 51% for humans. Its habit of recovering from a leap survives near-random sampling. Chord fit only departs from human practice at T=2.0T = 2.0 (AUC 0.59).

Temperature changes how many risks the model takes much more than how it handles them.


Baseline: a context-less Markov chain

If the transformer's risks fit the chords because it actually uses the harmony, a generator that never sees the harmony should fail exactly where the transformer passes. To test that, we scored ImprovMachine's Markov baseline: two independent first-order chains, one over absolute pitch and one over rhythm, with no chords at all. We fit it on 372 WJazzD training solos, generated 22 solos of 32 bars, and laid solo ii over the changes of WJazzD val song ii. The judges and the human calibration are unchanged.

JJ can't tell the chain from a human player (AUC 0.48 in the Results table). A first-order pitch chain still follows a leap with ordinary-sized moves, which is what JJ rewards. Judged only on how it continues, the chain looks like it handles its risks.

Chord fit can. It flags 11.3% of Markov risks, against 3.5% for humans and 2.9% for the transformer (AUC 0.67). That's about the score of the "continuation ignores the harmony" corruption above (0.64), which is exactly what a chord-blind chain produces. On average, the notes around a Markov risk sit worse on the harmony than a harmony-blind guess (−0.47 bits). Around human and transformer risks they gain +0.36 and +0.42 bits. The chain's risks are random notes it happens to walk away from smoothly; the transformer's are notes the chords account for.

Swapping the chords confirms it. The table below shows how much the chord-aware judge BB gains over the melody-only judge AA (bits per note) when each solo is scored under its own changes, under another val song's changes, or with no chords:

SetOwn changesChanges of val song i+1i+1i+7i+7i+13i+13No chords
Human (WJazzD val)+0.074−0.006−0.032−0.049−0.027
Transformer, T=1.2T = 1.2+0.129+0.006+0.004+0.008+0.020
Markov, WJazzD fit+0.179+0.176+0.182+0.187+0.181

The transformer behaves like a human here. Its gain is real under its own changes and drops to about zero under anyone else's, so its lines are written against the specific chords it was given. The Markov "gain" doesn't move at all, even with no chords. It isn't harmonic: the melody-only judge is confidently wrong about these lines (IC 0.69 bits above its own entropy, against 0.05 for humans), so mixing in any broader distribution lowers the cost. The chord-specific part of the gain is +0.10 bits for humans, +0.12 for the transformer, and 0.00 for the Markov chain.

Both generators take risks and move on from them smoothly. Only the transformer's risks fit the harmony, and only its harmonic gain depends on which chords it's given. That is the difference between a risk that's justified by context and one that just happens to land.


Caveats

  • The risk measures compare against WJazzD, which is mostly bebop and post-bop. They measure closeness to that style, not musical quality: a novel but good risk unlike anything in the corpus would get flagged.
  • JJ can't tell a deliberate risk from a random note smoothed over afterwards, since both leave the same notes.
  • Chord fit is a weak signal (a real note gains only 0.18 bits on average), so it separates sets of risks, not single notes.
  • The generated sets are small: 17–105 risk events per temperature.
  • The Markov baseline is one simple, chord-blind generator. The judges are also miscalibrated on its lines, which inflates its risk rate (6.5 per 100 notes), so it is better compared by JJ and chord fit than by how many risks it takes.