pytorch / opacus / 14372116584

80%

push

github

Fix Fast Gradient Clipping bias gradient calculation for three dim data (#751)

Summary:
Pull Request resolved: https://github.com/pytorch/opacus/pull/751

The bias grad calculation for three dim data was incorect.

Let `G = g^Tg`, where `g`, of dimensions `Txd` be the per-sample activation gradient, where `T` is the number of tokens and `d` dimension.

The per-sample gradient norm  with respect to bias is
`vec(G)^T vec(1)`, instead of the erroneous,`vec(G)^T vec(G)` before. This diff fixes it.

Reviewed By: aparna-aketi, HuanyuZhang

Differential Revision: D70823094

fbshipit-source-id: c1fe1dd7f

15 of 15 new or added lines in 2 files covered. (100.0%)

5299 of 6188 relevant lines covered (85.63%)

1.91 hits per line