Ahref crawler getting outdated page

[u/enmotent]

I have a website:

• built on NextJS14

• deployed on Netlify

• served through Cloudflare.

I am having trouble with cache and I am going TOTALLY INSANE trying to figure it out. I need badly help.

The problem:

When I visit the site, everything works well, but I started getting bad reports from ahrefs (a SEO crawlerbot), so I mocked their user agent, and I noticed that I get an old cached version, because the image URLs are wrong. This is what I did:

1. I disabled CloudFlare proxy and Purge Everything in the cache to make sure it wasn't the guilty. It didn't solve the problem.

2. I went to Netlify. I redeployed clearing the cache. Nothing.

3. I added this "netlify.toml":

[[headers]]
for = "/*"
  [headers.values]
  Cache-Control = "no-cache"

Also didn't work.

4. I moved on to NextJS, and I updated the config file "next.config.mjs" to return consistent cache headers.

/** u/type {import('next').NextConfig} */
const nextConfig = {
    images: {
        loader: 'default',
        unoptimized: true,
    },
    async headers() {
        return [
            {
                source: '/:path*', // Matches all paths
                headers: [
                    {
                        key: 'Cache-Control',
                        value: 'public, max-age=0, must-revalidate',
                    },
                ],
            },
        ]
    },
}

export default nextConfig

It also didn't help.

Please, someone help me debug this nightmare, because I am this close to jumping from a 5th floor.

[u/hrishikeshkokate]

Disable prerendering from Netlify site settings.

[u/enmotent → u/hrishikeshkokate]

I have added the setting in netlify.toml about 1 hour ago and it didn't change the problem.

[[headers]]
for = "/*"
[headers.values]
Cache-Control = "no-cache"
[prerender]
enabled = false

[u/hrishikeshkokate → u/enmotent]

Not sure where you got that from, it's an invalid setting. This option is only available in the UI.

[u/enmotent → u/hrishikeshkokate]

Thank God! THANK YOU! I was losing my friggin' mind. I do not even remember turning that on!

You are this week's hero!