Attention, May I Have Your Decision? Localizing Generative Choices in Diffusion Models

Katarzyna Zaleska    Łukasz Popek    Monika Wysoczańska    Kamil Deja

CVPR 2026

Abstract

Text-to-image diffusion models routinely face ambiguous prompts that under-specify visual details, forcing them to make implicit decisions about unspecified attributes. We posit that this decision-making is computationally localized within the model's architecture rather than being uniformly distributed. We introduce a probing technique that identifies the layers exhibiting the highest attribute separability, and find that self-attention layers are the locus where these implicit choices are resolved. Building on this insight, we propose ICM (Implicit Choice-Modification), a targeted steering method that modifies only the identified self-attention layers to control the model's implicit decisions. Experiments demonstrate that ICM achieves superior debiasing performance with fewer artifacts than existing approaches, providing both a new tool for controllable generation and a better understanding of where generative choices live in diffusion models.

BibTeX

@inproceedings{zaleska2026icm,
  title={Attention, May I Have Your Decision? Localizing Generative Choices in Diffusion Models},
  author={Zaleska, Katarzyna and Popek, {\L}ukasz and Wysocza{\'n}ska, Monika and Deja, Kamil},
  booktitle={CVPR},
  year={2026}
}