bitshifter / glam-rs / 36407103371

99%

Build Type

push

github

Committed by web-flow

Commit Message

fix(mat): only return a finite inverse from try_inverse/inverse_or_zero (#880)

# Objective

`try_inverse()` and `inverse_or_zero()` could return an invalid
(inf/NaN) inverse for matrices that are invertible. Two cases:

- The determinant is subnormal, so `1.0 / det` overflows to infinity.
- The determinant and its reciprocal are finite, but a cofactor (a 2x2
minor for `Mat3`, a 3x3 minor for `Mat4`) overflows while it is
computed. For example `Mat3::from_diagonal(Vec3::new(1e20, 1e20,
1e-20))` has `det = 1e20` and `1 / det = 1e-20`, but the cofactor `1e20
* 1e20` overflows, so the resulting inverse entry is `inf`.

`project_onto()` and `reject_from()` had the same class of bug: `rhs *
self.dot(rhs) * (1.0 / rhs.dot(rhs))` produced `NaN` for a very small
(but non-zero) `rhs`.

## Solution

- `inverse()` keeps the reciprocal-multiply fast path, so performance is
unchanged, and asserts the resulting matrix is finite under
`glam_assert` (matching `normalize`).
- `inverse_checked::<true>` (used by `try_inverse()` and
`inverse_or_zero()`) checks `m.is_finite()` on the resulting matrix, so
`try_inverse()` returns `None` and `inverse_or_zero()` returns `ZERO`
whenever the inverse is not finite. This catches both cases above.
- `project_onto()` and `reject_from()` compute `self.dot(rhs) /
rhs.dot(rhs)` directly, which is correct for tiny `rhs` and uses one
fewer vector multiply.

## Testing and linting

- Added tests for both overflow cases (subnormal determinant and
cofactor overflow) for `Mat2`/`Mat3`/`Mat3A`/`Mat4` (f32 + f64),
asserting `try_inverse().is_none()`, `inverse_or_zero() == ZERO` and
`should_glam_assert!({ m.inverse() })`.
- Added a test that `project_onto()` of a tiny vector is finite.

Coverage Stats

162 of 162 new or added lines in 29 files covered. (100.0%)

16 existing lines in 3 files now uncovered.

93332 of 94266 relevant lines covered (99.01%)

123370.64 hits per line