pytorch / opacus / 12970029985

80%

Build Type

push

github

Committed by facebook-github-bot

Commit Message

Modifying DPLossFastGradientClipping to add support for generative tasks with ghost clipping (#722)

Summary:
Pull Request resolved: https://github.com/pytorch/opacus/pull/722

Generative tasks for NLP output predictions of shape (B,T,C) i.e., (batch_size, sequence_length, vocab_size). To compute the cross-entropy loss in this case, usually the predictions are reshaped to (BxT, C) and targets to (BxT). This creates an issue with Ghost Clipping per sample loss computation as BxT is seen as the batch_size. In particular, the current implementation of Ghost Clipping results in loss_per_sample, coeff variables to have a shape of BxT and B respectively. This causes a shape mismatch error. This diff fixes that error by collapsing the loss_per_sample variable to shape B i.e., the loss across the sequence_length dim is averaged/summed.

Reviewed By: EnayatUllah

Differential Revision: D68047256

fbshipit-source-id: ad7614e2c

Run Details

4 of 10 new or added lines in 1 file covered. (40.0%)

1 existing line in 1 file now uncovered.

5099 of 5869 relevant lines covered (86.88%)

1.94 hits per line