Transformer Meets Diffusion: How the Transfusion Architecture Empowers GPT-4o’s Creativity

OpenAI’s GPT-4o represents a new milestone in multimodal AI: a single model capable of generating fluent text and high-quality images in the same output sequence. Unlike previous systems (e.g., ChatGPT) that had to invoke an external image generator like DALL-E, GPT-4o produces images natively as part of its response. This advance is powered by a novel Transfusion architecture described in 2024 by researchers at Meta AI, Waymo, and USC. Transfusion marries the Transformer models used in language generation with the Diffusion models used in image synthesis, allowing one large model to handle text and images seamlessly. In GPT-4o, the language model can decide on the fly to generate an image, insert it into the output, and then continue generating text in one coherent sequence. 

Let’s look into a detailed, technical exploration of GPT-4o’s image generation capabilities through the lens of the Transfusion architecture. First, we review how Transfusion works: a single Transformer-based model can output discrete text tokens and continuous image content by incorporating diffusion generation internally. We then contrast this with prior approaches, specifically, the tool-based method where a language model calls an external image API and the discrete token method exemplified by Meta’s earlier Chameleon (CM3Leon) model. We dissect the Transfusion design: special Begin-of-Image (BOI) and End-of-Image (EOI) tokens that bracket image content, the generation of image patches which are later refined in diffusion style, and the conversion of these patches into a final image via learned decoding layers (linear projections, U-Net upsamplers, and a variational autoencoder). We also compare empirical performance: Transfusion-based models (like GPT-4o) significantly outperform discretization-based models (Chameleon) in image quality and efficiency and match state-of-the-art diffusion models on image benchmarks. Finally, we situate this work in the context of 2023–2025 research on unified multimodal generation, highlighting how Transfusion and similar efforts unify language and image generation in a single forward pass or shared tokenization framework.

From Tools to Native Multimodal Generation  

Prior Tool-Based Approach: Before architectures like GPT-4o, if one wanted a conversational agent to produce images, a common approach was a pipeline or tool-invocation strategy. For example, ChatGPT could be augmented with a prompt to call an image generator (such as DALL·E 3) when the user requests an image. In this two-model setup, the language model itself does not truly generate the image; it merely produces a textual description or API call, which an external diffusion model renders into an image. While effective, this approach has clear limitations: the image generation is not tightly integrated with the language model’s knowledge and context.

Discrete Token Early-Fusion: An alternative line of research made image generation endogenously part of the sequence modeling by treating images as sequences of discrete tokens. Pioneered by models like DALL·E (2021), which used a VQ-VAE to encode images into codebook indices, this approach allows a single transformer to generate text and image tokens from one vocabulary. For instance, Parti (Google, 2022) and Meta’s Chameleon (2024) extend language modeling to image synthesis by quantizing images into tokens and training the model to predict those tokens like words. The key idea of Chameleon was the “early fusion” of modalities: images and text are converted into a common token space from the start.

However, this discretization approach introduces an information bottleneck. Converting an image into a sequence of discrete tokens necessarily throws away some detail. The VQ-VAE codebook has a fixed size, so it may not capture subtle color gradients or fine textures present in the original image. Moreover, to retain as much fidelity as possible, the image must be broken into many tokens, often hundreds or more for a single image. This makes generation slow and training costly. Despite these efforts, there is an inherent trade-off: using a larger codebook or more tokens improves image quality but increases sequence length and computation, whereas using a smaller codebook speeds up generation but loses detail. Empirically, models like Chameleon, while innovative, lag behind dedicated diffusion models in image fidelity.

The Transfusion Architecture: Merging Transformers with Diffusion  

Transfusion takes a hybrid approach, directly integrating a continuous diffusion-based image generator into the transformer’s sequence modeling framework. The core of Transfusion is a single transformer model (decoder-only) trained on a mix of text and images but with different objectives for each. Text tokens use the standard next-token prediction loss. Image tokens, continuous embeddings of image patches, use a diffusion loss, the same kind of denoising objective used to train models like Stable Diffusion, except it is implemented within the transformer.

Unified Sequence with BOI/EOI Markers: In Transfusion (and GPT-4o), text and image data are concatenated into one sequence during training. Special tokens mark the boundaries between modalities. A Begin-of-Image (BOI) token indicates that subsequent elements in the sequence are image content, and an End-of-Image (EOI) token signals that the image content has ended. Everything outside of BOI…EOI is treated as normal text; everything inside is treated as a continuous image representation. The same transformer processes all sequences. Within an image’s BOI–EOI block, the attention is bidirectional among image patch elements. This means the transformer can treat an image as a two-dimensional entity while treating the image as a whole as one step in an autoregressive sequence.

Image Patches as Continuous Tokens: Transfusion represents an image as a small set of continuous vectors called latent patches rather than discrete codebook tokens. The image is first encoded by a variational autoencoder (VAE) into a lower-dimensional latent space. The latent image is then divided into a grid of patches, & each patch is flattened into a vector. These patch vectors are what the transformer sees and predicts for image regions. Since they are continuous-valued, the model cannot use a softmax over a fixed vocabulary to generate an image patch. Instead, image generation is learned via diffusion: The model is trained to output denoised patches from noised patches.

Lightweight modality-specific layers project these patch vectors into the transformer’s input space. Two design options were explored: a simple linear layer or a small U-Net style encoder that further downsamples local patch content. The U-Net downsampler can capture more complex spatial structures from a larger patch. In practice, Transfusion found that using U-Net up/down blocks allowed them to compress an entire image into as few as 16 latent patches with minimal performance loss. Fewer patches mean shorter sequences and faster generation. In the best configuration, a Transfusion model at 7B scale represented an image with 22 latent patch vectors on average.

Denoising Diffusion Integration: Training the model on images uses a diffusion objective embedded in the sequence. For each image, the latent patches are noised with a random noise level, as in a standard diffusion model. These noisy patches are given to the transformer (preceded by BOI). The transformer must predict the denoised version. The loss on image tokens is the usual diffusion loss (L2 error), while the loss on text tokens is cross-entropy. The two losses are simply added for joint training. Thus, depending on its current processing, the model learns to continue text or refine an image.

At inference time, the generation procedure mirrors training. GPT-4o generates tokens autoregressively. If it generates a normal text token, it continues as usual. But if it generates the special BOI token, it transitions to image generation. Upon producing BOI, the model appends a block of latent image tokens initialized with pure random noise to the sequence. These serve as placeholders for the image. The model then enters diffusion decoding, repeatedly passing the sequence through the transformer to progressively denoise the image. Text tokens in the context act as conditioning. Once the image patches are fully generated, the model emits an EOI token to mark the end of the image block.

Decoding Patches into an Image: The final latent patch vectors are converted into an actual image. This is done by inverting the earlier encoding: first, the patch vectors are mapped back to latent image tiles using either a linear projection or U-Net up blocks. After this, the VAE decoder decodes the latent image into the final RGB pixel image. The result is typically high quality and coherent because the image was generated through a diffusion process in latent space.

Transfusion vs. Prior Methods: Key Differences and Advantages  

Native Integration vs. External Calls: The most immediate advantage of Transfusion is that image generation is native to the model’s forward pass, not a separate tool. This means the model can fluidly blend text and imagery. Moreover, the language model’s knowledge and reasoning abilities directly inform the image creation. GPT-4o excels at rendering text in images and handling multiple objects, likely due to this tighter integration.

Continuous Diffusion vs. Discrete Tokens: Transfusion’s continuous patch diffusion approach retains much more information and yields higher-fidelity outputs. The transformer cannot choose from a limited palette by eliminating the quantization bottleneck. Instead, it predicts continuous values, allowing subtle variations. In benchmarks, a 7.3B-parameter Transfusion model achieved an FID of 6.78 on MS-COCO, compared to an FID of 26.7 for a similarly sized Chameleon model. Transfusion also had a higher CLIP score (0.63 vs 0.39), indicating better image-text alignment.

Efficiency and Scaling: Transfusion can compress an image into as few as 16–20 latent patches. Chameleon might require hundreds of tokens. This means that the transfusion transformer takes fewer steps per image. Transfusion matched Chameleon’s performance using only ~22% of the compute. The model reached the same language perplexity using roughly half the compute as Chameleon.

Image Generation Quality: Transfusion generates photorealistic images comparable to state-of-the-art diffusion models. On the GenEval benchmark for text-to-image generation, a 7B Transfusion model outperformed DALL-E 2 and even SDXL 1.0. GPT-4o renders legible text in images and handles many distinct objects in a scene.

Flexibility and Multi-turn Multimodality: GPT-4o can handle bimodal interactions, not just text-to-image but image-to-text and mixed tasks. For example, it can show an image and then continue generating text about it or edit it with further instructions. Transfusion enables these capabilities naturally within the same architecture.

Limitations: While Transfusion outperforms discrete approaches, it still inherits some limitations from diffusion models. Image output is slower due to multiple iterative steps. The transformer must perform double duty, increasing training complexity. However, careful masking and normalization enable training to billions of parameters without collapse.

Related Work and Multimodal Generative Models (2023–2025)  

Before Transfusion, most efforts fell into tool-augmented models and token-fusion models. HuggingGPT and Visual ChatGPT allowed an LLM to call various APIs for tasks like image generation. Token-fusion approaches include DALL·E, CogView, and Parti, which treat images as sequences of tokens. Chameleon trained on interleaved image-text sequences. Kosmos-1 and Kosmos-2 were multimodal transformers aimed at understanding rather than generation.

Transfusion bridges the gap by keeping the single-model elegance of token fusion but using continuous latent and iterative refinement like diffusion. Google’s Muse and DeepFloyd IF introduced variations but used multiple stages or frozen language encoders. Transfusion integrates all capabilities into one transformer. Other examples include Meta’s Make-A-Scene and Paint-by-Example, Stability AI’s DeepFloyd IF, and HuggingFace’s IDEFICS.

In conclusion, the Transfusion architecture demonstrates that unifying text and image generation in one transformer is possible. GPT-4o with Transfusion generates images natively, guided by context and knowledge, and produces high-quality visuals interleaved with text. Compared to prior models like Chameleon, it offers better image quality, more efficient training, and deeper integration.

Sources

• https://openai.com/index/introducing-4o-image-generation/
• https://arxiv.org/pdf/2102.12092
• https://ar5iv.labs.arxiv.org/html/2405.09818
• https://arxiv.org/pdf/2408.11039v1
• https://arxiv.org/pdf/2206.10789
• https://ai.meta.com/blog/generative-ai-text-images-cm3leon/
• https://github.com/deep-floyd/IF

Also, feel free to follow us on Twitter and don’t forget to join our 85k+ ML SubReddit.

read more