From 8-Bit Sprites to Ray Tracing: How Game Graphics Became Almost Unrecognizable

Take a screenshot from a modern game and place it beside something released in 1980.

They barely appear to belong to the same medium.

One may contain realistically shaded skin, enormous landscapes, volumetric fog, physically based materials, dynamic shadows and reflections calculated from simulated rays of light. The other may represent a character using fewer colored blocks than a modern desktop icon.

It is tempting to describe the forty-plus years between them simply as "computers became faster."

That is true, but unsatisfying.

The history of game graphics is really a history of developers repeatedly discovering convincing shortcuts, then abandoning some of those shortcuts when hardware finally became powerful enough to do something closer to the real thing.

Ray tracing is merely the latest chapter.

Early games had almost nothing to work with

The first generations of game hardware operated under restrictions that seem almost absurd today.

Memory was scarce.

Processors were slow.

Storage was tiny.

Graphics hardware could display only limited colors and relatively small numbers of moving objects.

Developers therefore communicated through symbols.

A handful of pixels became a spaceship.

A square became a ball.

A tiny human figure became recognizable because the player cooperated with the illusion.

This is one of the enduring strengths of pixel art: the brain willingly fills in missing information.

Early game graphics were not failed attempts at realism.

They were visual languages adapted to extreme technical constraints.

Sprites allowed the world to move

One of the central ideas in 2D graphics was the sprite: a movable image that hardware could position independently over a background.

Sprites were perfect for games.

A background could represent the level while separate graphical objects represented the player, enemies, bullets and effects.

Specialized console hardware became extremely good at moving these objects around.

Developers learned tricks for recycling sprite graphics, changing palettes, flipping artwork and constructing large characters from several smaller sprites.

What looked like one large enemy might actually be assembled from many individual pieces.

The limitations became part of the aesthetic.

Developers started faking 3D long before consoles could properly render it

Players wanted depth.

Hardware often could not provide full polygonal 3D.

So developers cheated.

Racing games scaled roadside objects as they approached the camera.

Pseudo-3D shooters represented environments using mathematical shortcuts.

Isometric games arranged 2D graphics to resemble three-dimensional spaces.

Pre-rendered graphics used powerful workstations to create detailed 3D objects ahead of time, then converted the results into ordinary 2D images the console could display cheaply.

These techniques produced games that looked increasingly three-dimensional without requiring the machine to build a complete 3D world in real time.

Doom is a famous transitional example. Its environments create convincing three-dimensional movement while relying on significant architectural restrictions compared with later fully polygonal engines.

The illusion was good enough to change the industry.

Then polygons took over

A polygonal game describes objects as geometry—usually networks of triangles positioned in three-dimensional space.

Instead of storing a character purely as a collection of 2D frames, the game can define a 3D model and render it from different angles.

That creates extraordinary flexibility.

The camera can move freely.

Characters can rotate.

Lighting can respond to geometry.

The same environment can be viewed from many positions.

The cost is computation.

The machine must transform enormous numbers of vertices, determine what should be visible and turn three-dimensional geometry into pixels on a two-dimensional screen.

Early 3D consoles therefore produced worlds that now look wonderfully angular. Characters had pointed faces. Circular objects became obvious polygons. Textures stretched across simple geometry.

At the time, it looked like the future.

Rasterization became the workhorse of real-time 3D

Most traditional real-time 3D graphics rely heavily on rasterization.

At a high level, the engine takes geometric shapes—usually triangles—projects them onto the screen and determines which pixels they cover.

Rasterization became enormously efficient on graphics hardware and remains central to game rendering today. NVIDIA's technical explanation describes rasterization as the longstanding technique for displaying 3D objects on a 2D screen, while noting that real-time ray tracing historically demanded far more computation.

The challenge is that simply drawing triangles does not automatically produce realistic light.

Developers spent decades inventing clever ways to fake it.

Almost everything you thought was "lighting" contained tricks

A mirror is expensive if you genuinely simulate what light does.

So games can render another copy of the scene.

Soft shadows are expensive.

Use shadow maps.

Indirect light bouncing around a room is expensive.

Calculate lighting in advance and store the result in lightmaps.

Fine surface detail is expensive to model geometrically.

Fake it with normal maps.

Distant geometry is expensive.

Replace it with simpler models.

Realistic reflections are expensive.

Use screen-space information, environment maps or cubemaps.

None of these are dishonest techniques. They are the heart of real-time graphics.

A great graphics programmer is often somebody who can produce the perceptual result of an expensive physical process without paying its full computational price.

Programmable shaders changed what surfaces could be

Early 3D hardware used comparatively rigid pipelines.

As GPUs became programmable, developers gained much more control over how vertices and pixels were processed.

Shaders could describe sophisticated materials and effects directly.

Metal could respond differently from skin.

Water could distort the world.

Normal mapping could make a relatively simple surface appear covered in fine geometric detail.

Post-processing could add color grading, depth of field, bloom and numerous camera-like effects.

NVIDIA's retrospective divides modern graphics history around this transition to programmable shading and the later move toward hybrid ray-traced rendering.

Games did not suddenly become realistic.

Developers simply received a much larger box of illusions.

Physically based rendering made materials behave more consistently

As games became visually complex, manually tweaking every object became increasingly difficult.

Physically based rendering, or PBR, gave artists material systems inspired more directly by how real surfaces interact with light.

A metal surface should not merely be "gray and shiny."

Its response depends on physical properties.

Rough material scatters reflections differently from polished material.

Artists can describe those characteristics in textures and allow the lighting system to treat materials consistently across environments.

This is one reason modern games can maintain convincing surfaces across dramatically different lighting conditions.

And eventually we came back to rays

Ray tracing itself is not new.

Researchers understood the fundamental concept decades before GPUs could use it effectively in real-time games.

Imagine tracing a ray from the camera through a pixel and determining what object it encounters. Additional rays can test lights, reflections and other interactions.

The technique can produce extremely convincing images because it models aspects of light propagation more directly.

The problem was cost.

Movies could spend seconds, minutes or much longer calculating frames offline.

Games need another frame almost immediately.

NVIDIA researcher Morgan McGuire has described real-time ray tracing as an idea whose practical use in consumer games had to wait decades for sufficient computing performance.

That threshold began changing significantly with hardware acceleration for ray tracing in consumer GPUs.

Modern games usually don't throw rasterization away

This is an important misconception.

A ray-traced game is not necessarily abandoning decades of rasterization and calculating every pixel through pure ray tracing.

Many modern engines are hybrid renderers.

Rasterization handles much of the image extremely efficiently.

Ray tracing is added where it provides particular benefits: reflections, shadows, ambient lighting or global illumination.

NVIDIA similarly characterizes the modern transition as one of ray tracing and rasterization coexisting rather than rasterization instantly disappearing.

The old technique remains valuable because it is extraordinarily fast.

Path tracing pushes the idea much further

Path tracing uses much more extensive ray simulation to approximate how light bounces through a scene.

The results can be spectacular.

They are also expensive.

Modern games experimenting with path-traced modes rely heavily on GPU acceleration, denoising and increasingly sophisticated image reconstruction to produce usable frame rates.

Interestingly, this brings gaming graphics back to the central theme that has existed since the beginning:

Do as little work as possible while convincing the player's eyes that you did much more.

The tricks changed.

The philosophy didn't.

We never actually stopped faking graphics

The phrase "photorealistic graphics" can create the impression that game engines are gradually becoming literal simulations of reality.

They aren't.

Real-time graphics remain an enormous collection of approximations.

Textures represent detail that is not physically modeled.

Level-of-detail systems replace distant geometry.

Upscaling algorithms reconstruct pixels that were not originally rendered at full resolution.

Denoisers estimate the clean result of ray-traced lighting from a limited sample.

Animation systems approximate human motion.

Physics engines approximate physical behavior.

Even today's most spectacular game is a carefully constructed lie performed dozens or hundreds of times every second.

The difference between Pong and modern ray-traced graphics is therefore not that games eventually stopped cheating.

Developers simply became astonishingly good at it.

And perhaps that is the most impressive part of the entire history.

Every generation receives more computing power.

Every generation immediately finds something impossibly expensive that it wants to render.

Then developers begin inventing shortcuts all over again.