We run this site on Next.js and Vercel. It has about 7,600 URLs in the sitemap. Google had indexed 2,700 of them and left 4,790 alone — and 3,005 of those were sitting in Discovered – currently not indexed, which means Google found the URL and never actually fetched it.
That is the polite way of saying: you are publishing faster than we are crawling.
So we opened the Crawled – currently not indexed report to find out what the crawler was spending its time on instead.
The mistake we nearly shipped
Looking at the example URLs on screen, it seemed obvious. Non-English locales everywhere. Google was clearly skipping our translated pages, and the fix was to go and rework hreflang.
We were about a day of work away from doing exactly that.
Then we exported the CSV instead.
Out of 1,000 URLs in that bucket:
- 971 were
/_next/static/**or/favicon.ico - those 971 URLs pointed at only 222 distinct files
- so roughly 77% were duplicate crawls of assets Google already had
The locale theory was never in danger. It just happened to be what the top of the list looked like.
Here is why that happens, and it will happen to you too: the example URLs Search Console shows in its interface are sorted by last crawl date. Your newest real pages sit on top. The bulk of the bucket — the part you actually need to see — is below the fold, and you never scroll that far. Eyeball that list and you will generalise from a biased sample every time.
Export the CSV before you conclude anything.
What was actually happening
Vercel has a feature called Skew Protection. When you deploy while someone is mid-session, their browser may ask for a JavaScript chunk that no longer exists in the new build. Skew Protection solves that by appending ?dpl=<deploy-id> to every static asset URL, so a client keeps being served the build it started with.
It is a genuinely good feature and it fixes a real problem.
It also means every deploy mints a complete new set of URLs for byte-identical files. Googlebot has no way to know that main.js?dpl=aaa and main.js?dpl=bbb are the same thing. It sees two URLs, so it fetches two URLs.
By the time we looked, it had crawled 113 distinct dpl values. One font file, 39 times. The favicon, 37 times.
All of that came out of the same crawl budget those 3,005 real pages were queued in.
The fix, and the fix not to make
Turning Skew Protection off (Vercel dashboard → Settings → Advanced) stops new ones being minted. Asset filenames are already content-hashed, so caching is unaffected. The tradeoff you accept back is the original problem: a tab left open across a deploy can hit a chunk-load error.
For a content site with frequent deploys and short sessions, that is a trade worth making. For a long-session application it might not be.
What you should not do is "fix" this with Disallow: /_next/static in robots.txt. It is worse than the disease:
- Google needs your CSS and JavaScript to render the page and judge mobile usability. Block them and you hand it a broken page to evaluate.
- It would not drain the queue anyway. robots.txt stops the fetch, not the knowing. The URLs stay known, they simply get relabelled as blocked, and everything already queued stays queued.
Fix it at the source.
The part we did not expect
This turned out to be the third independent reason to deploy less often, after ISR write costs and build minutes.
We had been treating deploy frequency as a free variable — push whenever something is ready, several times a day. It is not free. Every deploy writes a fresh set of cache entries, burns build minutes, and now, it turns out, hands Googlebot a new pile of URLs that lead to files it already has.
If you are on Vercel with Next.js and more than a few hundred URLs, go and look at how many distinct dpl values Google has crawled. That number is your deploy count wearing a different hat — and unlike most SEO problems, it is entirely under your control.
The honest ending
It has been a few days since we turned it off. The old URLs do not 404 — a query parameter on a static asset still serves the file — so they do not vanish, they just stop being referenced from any rendered page. Our assumption is that Google deprioritises them once they stop appearing.
But we are not going to claim the queue has drained, because we do not know yet. Ask us in a month.
This is one of a few notes on how CalculatorAI is built. The same audit turned up a second problem — 21 of our 23 documentation pages were telling Google they were the homepage — which is a story for another post.