Genμ 2.0
Evaluation protocol
Three tracks with clearly defined evaluation protocols
Face and speech submissions are ranked only by ERB, the harmonic mean of erasure accuracy and retain accuracy. Preservation, quality, and intelligibility measurements are reported separately and do not affect leaderboard rank. Concept unlearning continues to use the original ERR protocol.
Common submission and evaluation contract
What participants may choose
- Tasks 1 and 2 are model-agnostic: any compatible face generator, TTS model, or voice-cloning model may be used.
- The unlearning algorithm, training schedule, parameter subset, adapter type, and auxiliary losses are open.
- Additional data may be used only when its source and role are documented in the repository.
What every submission must expose
- The exact original architecture and upstream checkpoint.
- The final unlearned weights or adapter and a script that loads them.
- Deterministic commands for preprocessing, unlearning, inference, and evaluation-output generation.
- All prompts, transcripts, identity references, seeds, camera controls, and model-specific conditioning required to reproduce the submitted outputs.
Submission revisions
Teams may revise a submission until 10 July 2026 by submitting the same track form again and marking it as a revision. The latest complete submission received before the deadline is treated as the official entry for that team and track.
Task 1: Face Identity Unlearning
Produce an unlearned face generator that forgets one specified person while preserving closely related identities, non-identity facial attributes, and general visual quality. See the home-page overview →
Problem statement
Given: an original face-generation model \(G_{\theta}\), its normal identity-conditioning interface, one target identity \(f\), and a retain set \(R_f\) of nearby identities.
Produce: an unlearned model \(G_{\theta'}\) or a clearly loadable adapter such that requests representing \(f\) no longer produce images that verify as the target identity, while requests for identities in \(R_f\) remain faithful and the output quality and requested non-identity attributes remain intact.
Identity conditioning may be model-specific: ArcFace embeddings, one or more reference images, learned identity tokens, latent identity codes, 3D-aware identity/camera codes, or another documented interface. Evaluation converts generated faces to a common ArcFace embedding space, so using a different conditioning mechanism does not change the identity-forgetting criterion.
Forget the target while preserving highly similar identities
The target and retain identities are intentionally close in the common ArcFace embedding space. The submitted method must remove only the target identity: target-conditioned generations should no longer verify as the target, while the provided neighboring identities should remain correctly recognizable and close to their original representations.
- Use the exact matched face inputs and inference settings defined above.
- Do not suppress a demographic group, collapse identity diversity, or degrade all generated faces.
- Forgetting and retention are measured across multiple references, prompts, samples, and seeds.
Change who is generated, not what was requested
The identity must be forgotten while non-identity requirements remain controllable. For matched prompts or conditions, attributes such as pose, expression, illumination, age-related appearance, smile, beard, eyewear, and camera/view settings should remain consistent whenever the chosen model supports them.
- Target outputs must fail identity verification.
- Requested attributes must remain present and measurable.
- Retained identities and overall image realism must not be sacrificed to obtain forgetting.
Face validation identity groups
Use each forget identity as the unlearning target and the identities in the same row as the retention neighborhood. These groups are public development cases; final evaluation may include additional held-out identities. Dataset access: CelebA official page. Teams using an FFHQ-compatible model may use the official FFHQ dataset tools.
| Validation set | Forget identity | Retain identities |
|---|---|---|
| Face Set 1 | 3422 | 5230, 5239, 1539 |
| Face Set 2 | 3376 | 3602, 608, 7405 |
Primary identity measurements
- Forget Accuracy (FA): percentage of target-conditioned generated samples that are still verified as the forgotten identity; lower is better.
- Erasure Accuracy (EA): \(EA=100-FA\); higher is better.
- Retain Accuracy (RA): percentage of retained-identity generated samples correctly verified as their intended identities; higher is better.
- ERB: harmonic balance of erasure and retention, so a method cannot rank highly by sacrificing either side.
Secondary preservation measurements
- Geometry Preservation (GP): \(100\bigl(1-\frac{\lVert\Delta D\rVert_F}{\lVert D_{\mathrm{orig}}\rVert_F+\epsilon}\bigr)\), clipped to \([0,100]\); higher is better.
- Attribute Retention (AR): percentage of requested or matched non-identity attributes preserved after unlearning; higher is better.
- Visual quality checks: FID and, where paired outputs are meaningful, SSIM/PSNR. These are reported separately for analysis and do not contribute to leaderboard rank.
Face Track ranking formula
The sole leaderboard ranking score is the Erasing–Retention Balance (ERB):
Prosody Preservation and Speech Attribute Retention are reported separately and are not combined with ERB.
Ranking rule: submissions are ordered only by higher \(\mathrm{ERB}_{\mathrm{face}}\). FA, RA, GP, AR, and visual-quality measurements are displayed as supporting diagnostics.
Variable and acronym glossary — Face Track
| \(G_{\theta}\) | Original face generator |
| \(G_{\theta'}\) | Submitted unlearned face generator |
| FA | Forget Accuracy: residual target-identity verification accuracy; lower is better |
| EA | Erasure Accuracy: \(100-FA\); higher is better |
| RA | Retain Accuracy over the specified retain identities; higher is better |
| ERB | Erasing–Retention Balance, the harmonic mean of EA and RA |
| GP | Normalized geometry-preservation score in \([0,100]\) |
| AR | Attribute Retention score in \([0,100]\) |
| FID / SSIM / PSNR | Separately reported visual-quality and fidelity checks |
Task 2: Speaker Identity Unlearning
Produce an unlearned TTS or voice-cloning model that forgets one target voice while preserving text, naturalness, prosody or requested style, and acoustically similar retain speakers. See the home-page overview →
Problem statement
Given: an original speech generator \(S_{\phi}\), its speaker-conditioning interface, one target speaker \(s_f\), a retain set \(R_s\), and evaluation transcripts and/or reference utterances.
Produce: an unlearned model \(S_{\phi'}\) or loadable adapter such that generated audio conditioned on \(s_f\) is no longer verified as the target speaker, while the transcript remains correct, speech remains natural, the requested prosodic/style information remains usable, and the voices in \(R_s\) remain recognizable.
Speaker conditioning may be model-specific: reference audio, speaker embeddings, speaker IDs, prompt tokens, or another documented interface. The repository must state sample rate, preprocessing, reference-audio duration, text normalization, decoding settings, and the exact checkpoint.
Forget one speaker while retaining acoustically similar speakers
The target and retain speakers form deliberately difficult, acoustically similar cohorts. The submitted method must remove the target-speaker identity while preserving the recognizability of the retain speakers. Prosody is evaluated as a secondary preservation property rather than being conflated with the identity-forgetting objective.
- Use the exact matched speech inputs and generation settings defined above.
- Do not obtain forgetting through global speaker collapse, monotone synthesis, broad accent suppression, or unintelligible audio.
- Forgetting and retention are measured across multiple transcripts, reference samples, and generation seeds.
Remove speaker identity, preserve communicative content and style
The target voice must be forgotten while intelligibility, naturalness, and supported attributes such as accent, emotion, speaking rate, rhythm, and expressive style remain stable. The goal is to remove who is speaking without unnecessarily changing what is said or how the requested utterance is delivered.
- Target outputs must fail speaker verification.
- ASR transcripts should remain accurate.
- Naturalness and requested attributes should remain comparable to the original model.
Speech validation speaker groups
Use each forget speaker as the unlearning target and the speakers in the same row as the retention cohort. The cohorts are same-gender clean-development groups intended to make retention more challenging. Final closeness is measured by the challenge speaker-embedding protocol, and final evaluation may include held-out speakers.
| Validation set | Forget speaker | Retain speakers |
|---|---|---|
| Speech Set 1 · female cohort | 84 | 5895, 3853, 5338 |
| Speech Set 2 · male cohort | 6241 | 6295, 2428, 3752 |
Primary speaker-identity measurements
- Forget Accuracy (FA): percentage of target-conditioned utterances still verified as the forgotten speaker; lower is better.
- Erasure Accuracy (EA): \(EA=100-FA\); higher is better.
- Retain Accuracy (RA): percentage of retained-speaker utterances correctly verified as their intended speakers; higher is better.
- ERB: harmonic balance of speaker erasure and retain-speaker accuracy.
Secondary preservation measurements
- Prosody Preservation Score (PPS): a \([0,100]\) score combining preservation of \(F_0\), energy, duration, and rhythm under the same transcript and decoding setup.
- Speech Attribute Retention (SAR): preservation of supported accent, emotion, speaking rate, and style attributes; higher is better.
- Quality and intelligibility checks: WER and MOS/UTMOS are reported separately for analysis and do not contribute to leaderboard rank.
Speech Track ranking formula
The sole leaderboard ranking score for the Speech Track is:
Prosody Preservation and Speech Attribute Retention are reported separately and are not combined with ERB.
Ranking rule: submissions are ordered only by higher \(\mathrm{ERB}_{\mathrm{speech}}\). FA, RA, PPS, SAR, WER, and naturalness measurements are displayed as supporting diagnostics.
Variable and acronym glossary — Speech Track
| \(S_{\phi}\) | Original speech generator |
| \(S_{\phi'}\) | Submitted unlearned speech generator |
| FA | Forget Accuracy: residual target-speaker verification accuracy; lower is better |
| EA | Erasure Accuracy: \(100-FA\); higher is better |
| RA | Retain Accuracy over the specified retain speakers; higher is better |
| ERB | Erasing–Retention Balance, the harmonic mean of EA and RA |
| PPS | Prosody Preservation Score in \([0,100]\) |
| SAR | Speech Attribute Retention score in \([0,100]\) |
| WER | Word Error Rate, reported as an intelligibility check |
| MOS / UTMOS | Human or model-predicted naturalness checks |
| ECAPA-TDNN | Common speaker-embedding evaluator used for cross-model comparison |
Task 3: Visual Concept Unlearning
Retain the original Genμ benchmark: erase a target concept from Stable Diffusion v1.4 while preserving unrelated and semantically adjacent concepts and resisting prompt-based recovery. See the home-page overview →
Problem statement
Given: the Stable Diffusion v1.4 reference checkpoint, a target concept, direct target prompts, retained and adjacent concepts, and indirect/adversarial prompt sets.
Produce: an unlearned checkpoint or parameter delta that suppresses the target concept not only for direct naming, but also for indirect descriptions and adversarially constructed prompts. The model should continue to generate retained and adjacent concepts with quality comparable to the original model.
Disallowed shortcut: a submission does not satisfy the task if it globally damages text-image alignment, blocks broad prompt categories, outputs low-quality images for all inputs, or removes adjacent concepts together with the target.
Primary metric: ERR Score (Erasing-Retention-Robustness), the harmonic mean of five sub-scores:
Tie-breaker — Weight-Change Ratio: \(\displaystyle \frac{1}{N_c}\sum \frac{\lVert\theta_{\mathrm{orig}}-\theta_{\mathrm{un}}\rVert}{\text{total parameters}}\). Smaller indicates fewer unnecessary parameter changes.
| \(N_c\) | Number of target concepts |
| \(\theta_{\mathrm{orig}}\) | Original Stable Diffusion v1.4 parameters |
| \(\theta_{\mathrm{un}}\) | Submitted unlearned parameters |
| \(\bar{A}_{\mathrm{fgt}}\) | Forgetting score for target concepts |
| \(\bar{A}_{\mathrm{ret}}\) | Retention score for unrelated concepts |
| \(\bar{A}_{\mathrm{adj}}\) | Retention score for correlated or adjacent concepts |
Published starting points: ESD, Concept Ablation, Forget-Me-Not, and FADE. See the Baselines page for descriptions, papers, and code links.
How to interpret a strong submission
A strong submission is selective: it reduces target verification while preserving the intended retain identities or speakers across multiple inputs and seeds. For Tasks 1 and 2, the leaderboard is determined only by ERB, which already balances erasure and retention. Additional preservation, quality, and intelligibility measurements are reported separately to make the evaluation transparent.