ZIPP: Zero-shot Image Personalization from Personas

ECCV 2026

Get in touch with us at behavior-in-the-wild@googlegroups.com

Abstract

Text-to-image diffusion models are increasingly deployed in creative contexts, yet remain impersonal — optimized for aggregate aesthetics rather than individual taste. Human preferences are also pluralistic: the same person may want muted, nostalgic portraits but vibrant, saturated street photography. Existing methods need dense interaction histories or per-user fine-tuning, fail in cold-start settings, and collapse each user's context-dependent preferences into one static style. We introduce zero-shot image personalization from personas (ZIPP), which conditions generation on natural-language personas — concise descriptors of a user's identity, interests, and aesthetic sensibilities — with no user-specific data and no weight updates. An LLM roleplays the persona to rewrite prompts, steering a frozen diffusion model toward taste-aligned outputs.

1. Zero-shot persona conditioning. Prepending a verbalized persona and rewriting the prompt personalizes any frozen diffusion model, with no per-user adapters or interaction history.
2. Persona mining at scale. An inductive Graph Attention Network over a 23M-user Reddit interaction graph, trained with dual contrastive objectives that align graph structure with users' visual behavior, is verbalized by an MLLM into coherent natural-language personas.
3. ZIPBench. The first zero-shot image-personalization benchmark, pairing 1.5K users with graph-mined personas and 40K generated images.
4. Consistent gains. Across four benchmarks and 14 LLMs from five families, persona conditioning improves personalization by 13–20%, with frontier models gaining most; few-shot ZIPP matches or exceeds fine-tuned baselines trained on 100+ examples per user.
5. Pluralism and equity. ZIPP best preserves intra-user preference diversity (lowest CMMD, 0.16 vs. 0.55), and IPF-normalized evaluation against global population marginals shows it substantially reduces the subpopulation bias baked into existing methods.
6. Human preference. Raters prefer ZIPP 79% of the time over generic generation and over every fine-tuned baseline, with no user-specific training.

Method

ZIPP turns large-scale behavior into reusable, interpretable personas, then uses them to steer any diffusion model through prompt rewriting.

Key Results

We evaluate ZIPP across four benchmarks — ZIPBench and PIP (historical generations), MovieLens and RapidData (preference pairs) — against zero-shot, few-shot, and fine-tuned baselines. We report CLIPScore (CS) and PIGReward (PIG) for personalization quality, a demographically IPF-normalized aggregate, and CMMD for pluralistic diversity (lower is better).

Pluralistic Preference Alignment

Effective personalization must capture not only what a user prefers on average, but also how their preferences vary across contexts. Someone who likes muted, animated portraits may want bright, saturated street photography — they should not get the same look for every prompt. We formalize this as pluralistic alignment: a method should achieve high average personalization while preserving the distributional spread of a user's true visual preference space, measured by CMMD (CLIP Maximum Mean Discrepancy) between reference and generated images.

To make this concrete, we follow a single individual end to end — from her raw Reddit activity, to the persona ZIPP mines, to the pluralistic visual preferences that persona predicts.

Methods that collapse a user into a fixed style are the most susceptible to pluralistic misalignment. ViPer reuses the same style attributes across all prompts (worst CMMD, 0.55); DrUM anchors generation around a narrow slice of history (0.31); retrieval-based TV (0.25) collapses when the prompt pool is small. ZIPP achieves the lowest CMMD (0.16) while maintaining the highest alignment, because personas encode multi-faceted preference priors — aesthetics, topics, culture, and affect — that modulate generation contextually rather than anchoring on a fixed reference set.

Lower CMMD = better: closer to the user's true spread of visual preferences. ZIPP (0.16) is the clear winner; ViPer (0.55) is the worst.

Demographic Alignment

Preference datasets have historically over-sampled WEIRD (Western, Educated, Industrialized, Rich, Democratic) populations — up to 96% of behavioral-science participants, representing only ~12% of the global population. The same bias has crept into image-personalization benchmarks: most (PIP, HPD, Pick-a-Pic) do not even report demographic distributions, making bias impossible to audit. We use RapidData, where each preference pair is tagged with anonymized group-level demographics, and apply Iterative Proportional Fitting (IPF / raking) to reweight per-subgroup scores so the aggregate reflects true UN population marginals rather than the majority-skewed sample.

Because the evaluation itself is skewed, methods that maximize aggregate scores are the most exposed. Under IPF normalization, DrUM drops the most — collapsing from 75.0 (N. America) to 58.0 (Africa), a 22.7% regional gap, and degrading sharply for older (55+), Latin American, Middle Eastern, and manual-trade users. ZIPP stays within ~1% of its raw score and maintains ≥76.5 CLIPScore across every region, because the LLM reads age- and culture-relevant context directly from the persona regardless of training-data representation. Even zero-shot ZIPP matches or beats trained baselines on underrepresented subgroups. See the Demographic (IPF) tab above for the full raw-vs-IPF breakdown.

Human Evaluation

We ran a controlled study with 50 demographically diverse participants, building each person's natural-language persona from a short intake survey on demographics, profession, and hobbies (never visual preferences). Each participant made pairwise A/B choices over 20 prompts, yielding 1,000 annotations. Annotators consistently preferred persona-conditioned generations, and a separate expert study confirmed the quality of mined personas.

Persona quality assessment

In a second study, three independent expert annotators rated 100 mined personas (~1,000 annotations) on a 1–5 scale. Personas were judged accurate and coherent, with substantial inter-annotator agreement (Cohen's κ = 0.64).

Examples

ZIPP rewrites a single base prompt from each persona's perspective, steering the diffusion model toward taste-aligned outputs. Browse the examples below to see the persona, the base prompt, and the rewritten prompt.

Exploring the Reddit Behavior Graph

Personas are mined from a Reddit interaction graph spanning 23M users, 40K subreddits, and 682M posts and comments (2015–2022). Below: interactive views of where and who the activity comes from, and how persona attributes map to visual interests.

BibTeX

@article{si2026zipp,
      title={ZIPP: Zero-shot Image Personalization from Personas},
      author={SI, Harini and Singh, Somesh and Singla, Yaman Kumar and Doermann, David and Shah, Rajiv Ratn},
      journal={arXiv preprint arXiv:2606.08841},
      year={2026}
    }

---