What Is a Cache? Why Fast Systems Keep Copies of Data Close By
Learn what a cache is, how cached data improves performance, why cache hits are faster than source lookups, and what trade-offs caching introduces.
A cache is one of those ideas that sounds more complicated than it really is.
At its core, a cache is temporary high-speed storage used to keep data that is likely to be needed again. Instead of repeatedly fetching the same information from a slower source, the system keeps a nearby copy and checks that copy first.
That simple pattern appears almost everywhere in computing: inside processors, web browsers, databases, APIs, operating systems, and large distributed applications.
The basic flow is:
Request
↓
Check cache
↓
Hit? ── yes ──► return cached data
│
no
↓
Fetch from original source
↓
Return result
↓
Optionally store in cache
The reason caching works is equally simple. Reusing data you already have is often much faster than retrieving or recomputing it again.
A Cache Trades Freshness for Speed
Imagine an application needs to display the same product information thousands of times per minute.
Without caching, every request might go all the way to a database:
User → application → database → application → user
That works, but it forces the database to answer the same kinds of requests over and over.
With a cache, the application can keep a temporary copy of commonly requested data:
User → application → cache
│
└── product already here
If the cached value is still usable, the database does not need to be involved at all.
This reduces latency, because the data can be returned more quickly, and it reduces workload on the original system. Fewer database queries, API calls, disk reads, or network requests can mean lower infrastructure cost and better performance under load, which is the same responsiveness trade-off behind latency vs bandwidth.
The trade-off is that the cached copy is a copy. If the original data changes, the cache may briefly contain an older version.
That is the central tension in caching:
faster access versus fresher data.
A Cache Hit Is the Fast Path
When the requested data is already available in the cache, the system has a cache hit.
Suppose an application asks for product 42 and the cache already contains it:
Request: product 42
↓
Cache
↓
product 42 found
↓
Return immediately
The original source may never be contacted.
A cache hit is valuable because the cache is normally chosen specifically for faster access. It might live in memory rather than on disk, on the same machine rather than across the network, or closer to the user than the original server.
The performance improvement depends on what the cache is avoiding, a point that also shows up in Cloudflare’s cache explainer.
Avoiding a database query might save a few milliseconds. Avoiding a request to a distant external service could save far more. A CPU cache can save tiny fractions of a second, but those savings happen billions of times and matter enormously to processor performance.
Different caches operate at very different scales, but they all exploit the same idea: keep useful data closer to where it will be needed next.
A Cache Miss Goes Back to the Source
If the requested item is not in the cache, the result is a cache miss.
The system then needs to retrieve or calculate the data through the slower path.
For example:
Request product 42
↓
Check cache
↓
Not found
↓
Query database
↓
Receive product 42
↓
Store in cache
↓
Return result
The first request still pays the full cost of retrieving the data.
The benefit appears on later requests. If product 42 is requested again before the cached copy expires or is removed, the system can return it directly from the cache.
This is why caching works best when data is reused.
If every request asks for completely different information that is never requested again, filling a cache may provide little benefit. Good caching depends on locality: recently used or frequently used data is often likely to be used again.
Caching Reduces More Than Response Time
Speed is the obvious benefit, but caching can improve an entire system.
Consider a popular page that requires ten database queries every time it loads. If one million people open that page, those queries can create enormous database pressure.
If the result can be cached, many requests may avoid those queries completely.
That means caching can reduce:
- database workload;
- network traffic;
- disk access;
- calls to external APIs;
- CPU spent recomputing the same result;
- pressure on backend services.
This can also improve scalability. A server that would otherwise handle 1,000 expensive requests per second may be able to serve much more traffic if most of those requests become inexpensive cache hits.
Caching therefore does not merely make one request faster. It can prevent downstream systems from doing unnecessary work.
Browser Caches Avoid Downloading the Same Files Repeatedly
One of the most familiar examples is the browser cache.
Web pages frequently reuse the same resources: images, stylesheets, JavaScript files, fonts, and other assets. Downloading every file again on every page visit would waste bandwidth and slow the experience, especially on pages built from HTML that reference the same supporting files repeatedly.
The browser can keep local copies.
Suppose a website includes:
/logo.png
/styles.css
/app.js
After downloading those files once, the browser may reuse them on later requests if its caching rules say the stored copies are still valid.
Instead of:
Browser → internet → server → download logo again
the browser may simply use:
Browser → local cache → logo
That can make repeat page loads noticeably faster.
Web caching is controlled through mechanisms such as HTTP cache headers, which tell browsers and intermediary caches how long a response may be reused and when it needs to be revalidated, as described in MDN’s HTTP caching guide.
This also explains why developers sometimes encounter the opposite problem: they change a file, refresh the page, and still see the old version because a cached copy is being reused.
CPU Caches Solve the Same Problem at a Much Smaller Scale
Processors use caching too.
A CPU can perform operations much faster than it can retrieve arbitrary data from main memory. If the processor had to wait for RAM every time it needed a value, a great deal of computing capacity would sit idle.
Modern processors therefore contain very fast CPU caches, typically arranged in levels such as L1, L2, and L3.
Frequently or recently accessed data can be copied closer to the processor cores.
The broad idea is still:
CPU needs data
↓
Check fastest cache
↓
If found → use it
If not → look farther away
The differences are mostly scale and implementation. A browser cache might hold files for minutes, days, or longer. A CPU cache may hold tiny pieces of data for extremely short periods while operating at hardware speeds.
Both are examples of the same engineering principle: the farther away the original data is, the more valuable a nearby reusable copy can become.
Applications Often Use Memory Caches
Applications frequently cache data directly in memory.
Imagine an API that needs a list of supported countries on almost every request. The list changes only occasionally, so repeatedly querying the database would be wasteful.
The application can load the list into memory and reuse it.
Because RAM access is generally much faster than performing an external database query, the improvement can be significant.
This might be an in-process cache, where each application instance keeps its own copy:
App instance A → local memory cache
App instance B → local memory cache
That approach is very fast, but it introduces a coordination problem. Each instance has a separate copy, so one may contain newer data than another.
Large applications therefore sometimes use a shared distributed cache, such as a dedicated in-memory caching service like Redis.
Then multiple application instances can access the same cache:
App A ─┐
App B ─┼──► shared cache
App C ─┘
The shared cache adds a network hop, so it is slower than local memory, but it can still be much faster than querying the original database and easier to keep consistent across many servers.
Databases Have Their Own Caching Layers
Databases also rely heavily on caching.
Frequently accessed pages, indexes, query plans, and other data may be kept in memory so the database does not repeatedly read them from slower storage.
Applications can also place a cache in front of a database.
For example:
Application
↓
Cache
↓ miss
Database
This is common when certain records are read far more often than they are changed.
Product details, configuration, user preferences, session information, and computed results can all be good cache candidates depending on the application.
But database caching introduces one of the hardest questions in system design:
What happens when the database changes but the cache still contains the old value?
That is the cache invalidation problem.
Cached Data Can Become Stale
Suppose a product costs $50.
The application caches:
product:42
price: $50
Later, someone changes the price in the database to $40.
If nothing updates or removes the cached value, users may continue seeing $50.
The cache is now stale.
This does not mean caching is broken. It means the system needs a policy for deciding how long old data is acceptable and how cached values are refreshed.
Some data can tolerate a little staleness. A news article view count being a few seconds behind may not matter.
Other data is far more sensitive. Serving an outdated permission, account balance, inventory quantity, or fraud decision may be unacceptable.
The correct caching policy therefore depends on the meaning of the data, not merely on its technical format.
Expiration Lets Old Entries Disappear Automatically
One simple way to limit staleness is to give cached entries an expiration time, often called a TTL, or time to live.
A cached value might be stored for:
60 seconds
After that time, the cache considers it expired.
The next request becomes a miss, forcing the system to retrieve fresh data and potentially cache it again.
This creates a cycle:
Fetch data
↓
Cache for 60 seconds
↓
Serve repeated requests
↓
Entry expires
↓
Fetch fresh data
TTL-based caching is simple and robust, but it accepts a defined period of possible staleness.
If the original data changes one second after being cached with a ten-minute TTL, the old version could theoretically be served for almost ten more minutes.
That may be perfectly acceptable for some data and completely unacceptable for others.
Invalidation Removes Data When the Source Changes
Instead of waiting for expiration, a system can actively invalidate or update the relevant cache entry when data changes.
Suppose an application updates product 42.
It might perform:
Update database
↓
Delete product:42 from cache
The next read misses the cache, fetches the new database value, and repopulates the entry.
Another strategy is to update the cached value directly.
This can provide fresher results than waiting for a TTL, but invalidation becomes harder as systems grow.
A single database change might affect several cached objects. Multiple application instances may have local caches. Events can fail to arrive. Network partitions can delay invalidation messages.
This is why cache invalidation has a reputation as a difficult engineering problem. Storing a copy is easy; knowing exactly when every copy has stopped being correct is much harder, especially in the kinds of architectures discussed in software architecture and system design.
Different Caching Strategies Change When the Cache Is Updated
Applications use several common patterns for coordinating caches with the original data source.
One common approach is cache-aside. The application checks the cache first and queries the database only on a miss. After retrieving the value, it places it in the cache.
That is the pattern we have been using:
Read
↓
Cache
↓ miss
Database
↓
Populate cache
Another pattern is write-through caching, where updates are written through the cache to the underlying storage so the cache is kept current as part of the write path.
There is also write-behind, where data may be written to the cache first and persisted to slower storage later. This can improve write performance but introduces greater durability and synchronization concerns, a pattern outlined in the Azure cache-aside pattern guide.
No strategy is universally best. The choice depends on whether the system prioritizes simplicity, freshness, write speed, read speed, consistency, or fault tolerance.
A Cache Has Limited Space
Caches cannot keep everything forever.
A CPU cache is tiny compared with main memory. A browser cannot retain every file it has ever downloaded. An application cache with unlimited growth would eventually consume all available memory.
When a cache becomes full, something has to be removed. This is called eviction.
Different systems use different eviction policies. A common idea is to remove items that have not been used recently, because recently accessed data is more likely to be requested again.
Other policies may consider frequency, age, priority, or memory size.
This introduces another type of cache miss. The data may have been cached previously, but it was evicted before the next request.
A cache therefore has to answer three basic questions:
What should we store? How long should we keep it? What should we remove when space runs out?
Those decisions determine whether the cache actually improves performance.
Caching Everything Can Make a System Worse
Because caches make reads faster, it is tempting to cache as much as possible.
That can backfire.
Caching rarely used data consumes memory without producing many hits. Very short-lived data may expire before reuse. Highly sensitive data may create additional security concerns. Rapidly changing data can introduce consistency problems that outweigh the performance benefit.
The cache itself also becomes infrastructure that needs monitoring and capacity planning.
If every request depends on a shared cache and that cache fails, the original database can suddenly receive a massive increase in traffic. This is sometimes called a cache stampede or thundering-herd problem when many requests simultaneously miss and attempt to rebuild the same cached data, the same kind of repeated downstream pressure that distributed tracing can help surface in production.
A production cache therefore needs to be treated as part of the system architecture, not merely as a speed switch.
Good caching starts by identifying expensive operations that are repeated often enough for reuse to matter.
Cache Hits Are Useful, but Hit Rate Is Not the Whole Story
Teams often monitor the cache hit rate: the percentage of cache lookups that successfully find the requested data.
A high hit rate can indicate that the cache is serving useful work.
But a percentage alone can be misleading.
Caching thousands of inexpensive operations while repeatedly missing one extremely expensive query may produce a high hit rate without solving the real performance problem.
Likewise, a relatively modest hit rate may still be valuable if each hit avoids a costly remote request.
The better question is not simply:
How often do we hit the cache?
It is:
How much expensive work does the cache prevent?
Latency, backend load, memory consumption, eviction rates, and stale-data behavior all matter alongside hit rate, which is why observability guidance like OpenTelemetry’s documentation becomes relevant once caches are part of the architecture.
Clearing a Cache Forces the System Back to the Source
When cached data becomes incorrect or troubleshooting requires a fresh copy, the cache can be cleared.
This might mean deleting one entry:
delete cache["product:42"]
or flushing an entire cache.
Afterward, requests need to return to the original source until the cache fills again.
That is why clearing a large production cache can cause a sudden performance impact. A system that normally serves 95% of requests from memory may suddenly send almost all of them to a database.
Caches are therefore often warmed gradually or repopulated deliberately rather than being flushed casually.
The same basic effect happens in a browser. Clearing browser cache removes stored resources, so websites need to download those files again on the next visit.
“Clear the cache” can fix stale-data problems, but it also throws away the performance benefit the cache had accumulated.
The Cache Is a Copy, Not Usually the Source of Truth
One of the safest ways to reason about caching is to distinguish the cache from the authoritative source.
Suppose a database owns the official product record.
The cache contains:
temporary copy of product 42
If that copy disappears, the system can reconstruct it from the database.
That makes the cache disposable.
This is an important design property. If losing the cache permanently loses critical business data, then it is no longer functioning purely as a cache; it has become part of the system’s authoritative storage model.
Some architectures deliberately blur this distinction, particularly with write-behind systems, but doing so introduces much stronger durability requirements.
For ordinary caching, a useful rule is:
The system should know where the real data lives and what happens when the cached copy disappears.
Caching Is Really About Avoiding Repeated Work
It is easy to think of a cache as simply “fast storage,” but that misses the deeper idea.
What matters is not the storage medium by itself. What matters is that the system avoids repeating an expensive operation.
The expensive operation might be:
- downloading a file;
- querying a database;
- calling another service;
- reading from disk;
- calculating a complex result;
- retrieving data from slower memory.
The cache remembers the result so the system can reuse it.
That gives the whole process a simple loop:
Request
↓
Have a usable cached result?
│
├── yes → return it
│
└── no → do expensive work
↓
save result
↓
return it
The next request can then take the shorter path.
That is why caches appear at nearly every level of computing. The technologies differ enormously, but the optimization is the same.
A cache keeps a temporary copy of useful data close to where it is needed, turning repeated expensive requests into faster cache hits. The difficult part is not storing the copy—it is deciding when that copy is still safe to use and when it needs to expire, be invalidated, or be refreshed.