FEniCS / ffcx / 33252992871

85%

Build Type

push

github

Committed by web-flow

Commit Message

Stack codegen performance fixes: quadrature-loop split, tensor contraction, CSE, coordinate unrolling, LICM fusion (#865)

* Split quadrature loop into evaluation and contraction

The fused quadrature loop interleaves evaluation of the varying
quantities with the tensor contraction. The strided reads of the basis
tables in the contraction stop GCC vectorising the loop, and because
they share a loop body with the evaluation, math functions of the
spatial coordinate are left scalar: a P2 load vector with a
sin(pi x) sin(pi y) sin(pi z) source spends 64% of its time in scalar
__sin_fma, where the P1 kernel of the same form calls libmvec's
four-wide _ZGVdN4v_sin.

Store the fw intermediates in an array indexed by quadrature point and
contract in a second loop, leaving the evaluation loop free of the
strided accesses. On a P2 Poisson load vector (tetrahedra, 1.57M cells)
the kernel goes from 98.6 to 42 ns/cell and end-to-end RHS assembly in
DOLFINx from 0.183 s to 0.098 s, with bit-identical output. P1 and
matrix assembly are unchanged to within run-to-run noise.

Applies to single-index quadrature rules with fw intermediates; other
cases keep the fused loop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Group tensor contraction terms sharing an inner-loop factor

Loop-invariant code motion in the tensor contraction hoisted each term
of an entry separately, creating one temporary array and one
accumulation into the element tensor per term. A 3D Poisson operator
has one term per pair of derivative directions, so the inner loop
carried nine temporaries and nine read-modify-writes of A where three
suffice: terms that share the factor left in the inner loop differ only
in what was hoisted, so one temporary can hold their sum.

Group the terms of each entry by the factors that remain in the inner
loop, sum the hoisted parts into a single temporary per group, and emit
one accumulation per group. For P1 Poisson on tetrahedra the contracti... (continued)

Coverage Stats

364 of 389 new or added lines in 6 files covered. (93.57%)

2 existing lines in 2 files now uncovered.

4558 of 5335 relevant lines covered (85.44%)

0.85 hits per line