meta-pytorch / opacus / 24817925856
80%

DEFAULT BRANCH: main
Ran
Jobs 3
Files 138
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 79.8% (+0.3%) from 79.477%
24817925856

push

github

meta-codesync[bot] 
Non-wrapping mode for better Transformers compatibility (#794)

Summary:
## Types of changes

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Docs change / refactoring / dependency upgrade

## Motivation and Context / Related issue

The primary goal of this PR is to introduce a **Non-wrapping mode** (controller-based) to Opacus. This feature allows for the computation of per-sample gradients without wrapping the original `nn.Module` in a `GradSampleModule` subclass.

This change addresses critical compatibility issues with third-party libraries—most notably HuggingFace `Transformers` and `Accelerate`—which often rely on strict module hierarchies or perform type checks that are disrupted by standard Opacus wrappers.

**Key Technical Changes:**
- **`GradSampleHooks` Controller**: Introduced a new architecture where gradient sampling hooks are managed by a standalone `GradSampleHooks` object instead of being embedded in a `GradSampleModule` wrapper.
- **`PrivacyEngine` Updates**: Modified `make_private` and `make_private_with_epsilon` to support a `wrap_model` parameter (defaults to `True` for backward compatibility). When `wrap_model=False`, Opacus attaches hooks to the original model and returns it along with a `GradSampleHooks` controller.
- **Enhanced Loss Compatibility**: Updated `DPLossFastGradientClipping` to handle custom loss implementations that lack a `reduction` attribute (a common occurrence in the `transformers` library). The implementation now explicitly sets the expected reduction if missing and restores the previous state after the backward pass.
- **HuggingFace Trainer Integration**: Verified the new mode via successful small-scale training runs using the HuggingFace `Trainer` with Fast Gradient Clipping. A new example (`examples/huggingface_trainer.py`) is included to d... (continued)

435 of 533 new or added lines in 20 files covered. (81.61%)

4 existing lines in 4 files now uncovered.

6471 of 8109 relevant lines covered (79.8%)

1.76 hits per line

Uncovered Changes

Lines Coverage ∆ File
23
15.65
 -1.35% opacus/tests/multigpu_precision_test.py
21
27.41
 -0.59% opacus/tests/mixed_precision_test.py
12
26.09
 -1.44% opacus/tests/multigpu_gradcheck_test.py
9
71.93
 -10.57% opacus/grad_sample/utils.py
9
31.08
 -1.78% opacus/tests/multigpu_adaptive_clipping_test.py
7
95.45
 opacus/tests/grad_sample_hooks_test.py
5
89.39
 -1.35% opacus/grad_sample/gsm_base.py
4
52.94
 -7.06% opacus/grad_sample/gsm_no_op.py
3
30.16
 2.09% opacus/grad_sample/grad_sample_module_fast_gradient_clipping_fsdp.py
3
40.54
 3.87% opacus/grad_sample/grad_sample_module_fast_gradient_clipping_tp.py
1
80.62
 0.0% opacus/privacy_engine.py
1
99.28
 opacus/tests/grad_sample_hooks_fast_gradient_clipping_test.py

Coverage Regressions

Lines Coverage ∆ File
1
31.08
 -1.78% opacus/tests/multigpu_adaptive_clipping_test.py
1
26.09
 -1.44% opacus/tests/multigpu_gradcheck_test.py
1
15.65
 -1.35% opacus/tests/multigpu_precision_test.py
1
27.41
 -0.59% opacus/tests/mixed_precision_test.py
Jobs
ID Job ID Ran Files Coverage
1 run-3 - 24817925856.1 72
47.43
 GitHub Action Run
2 run-2 - 24817925856.2 137
79.6
 GitHub Action Run
3 run-1 - 24817925856.3 137
79.59
 GitHub Action Run
Source Files on build 24817925856