FEniCS / ffcx / 32303673965

85%

main: 85%

Pull #865

github

Fix mass-matrix regression: pick loop-nest order by term count

Bisected the ~2x mass_p2 regression found while benchmarking this
branch against main to a single commit within PR #853
(52e7fa07, "Nest contraction loops with the contiguous tensor index
innermost"): tested it in isolation with the other two #853 commits
absent, and it alone reproduces the full regression.

That commit's row-contiguous loop order is exactly what drives its
own large wins on poisson_p3/vector_poisson_p2/stokes_th, so reverting
it outright (tested) recovers mass_p2 but roughly halves those gains.
Neither a restrict-qualified local row pointer nor explicit
ivdep/unroll pragmas on the accumulation loop changed anything either
(tested directly on the generated mass_p2.c).

What actually distinguishes the losing case: mass_p2's entries each
sum exactly one term (a plain product, no derivatives), while every
form that wins big from the contiguous order sums multiple terms per
entry (one per spatial direction, from a gradient-gradient or
divergence pairing). Picking the loop order per block on that signal
-- already available at this point via the existing 'keep' dict, no
new bookkeeping -- gets both: mass_p2 recovers to parity with main
(1.06x-1.07x instead of 0.54x-0.55x, strict FP and fast-math alike),
and every other benchmarked form keeps its win. stokes_th improves
further still (2.4x-3.6x to 2.4x-4.5x), since its velocity-pressure
coupling blocks are themselves single-term and now pick up the same
fix.

Verified against all 7 forms benchmarked on PR #865, strict FP and
with -ffast-math, 3 interleaved rounds each; full test suite still
212 passed, 1 skipped.

276 of 318 new or added lines in 4 files covered. (86.79%)

4475 of 5270 relevant lines covered (84.91%)

0.85 hits per line