Notifying search engines about new and changed pages with IndexNow

When you publish a page, the moment search engines notice it depends largely on their own crawl schedule. A sitemap helps that schedule, but it is not a call; an engine reads the sitemap as often as it sees fit. IndexNow is an open protocol that turns this relationship around: you report new, changed or deleted addresses yourself, and the participating search engines share the notification with each other.

This article describes how we set up IndexNow on our own sites: publishing the key file from Flask, sending addresses in a single batch request, reading the response codes correctly and, most importantly, deciding what to report and when. We will also be explicit about where Google stands, because that is the misunderstanding we run into most often.

How the protocol works in brief

IndexNow has three parts. The first is a key: a string you generate yourself, made of letters, digits and dashes, between 8 and 128 characters long. The second is a key file where that key is published on your site: usually a plain text file at the root, at /<key>.txt, containing nothing but the key itself. The third is the submission request: an HTTP request that says which addresses have changed.

When a search engine receives a submission, it fetches the key file from your domain and checks that the key inside matches the key in the request. That way only someone who actually controls the domain can submit notifications for it. The key is not a secret password; it already sits in a public file. Its job is to prove ownership.

The engines taking part in the protocol include Bing, Yandex, Seznam and Naver; the current list is published on indexnow.org. Because a submission sent to one engine is shared with the other participants, sending to a single endpoint is enough. We use the shared endpoint at api.indexnow.org.

Publishing the key file from Flask

Instead of committing the key to the repository as a file, we keep it in an environment variable. The same code can then run on different sites with different keys, and changing the key does not require a new release. Generating a key with Python takes one line:

py -3.12 -c "import secrets; print(secrets.token_hex(16))"

This prints a 32-character hexadecimal string, which stays within the character set and length limit the protocol allows. The route that publishes the file looks like this:

import os

from flask import Flask, Response, abort

app = Flask(__name__)


@app.get("/<anahtar>.txt")
def indexnow_anahtari(anahtar):
    """INDEXNOW_KEY environment variable → /<key>.txt; 404 if unset or the name does not match."""
    k = (os.environ.get("INDEXNOW_KEY") or "").strip()
    if k and anahtar == k:
        return Response(k + "\n", mimetype="text/plain")
    abort(404)

Note two details. If the variable is not set, the endpoint stays completely closed, so random .txt addresses do not return 200 in a local development environment or on a misconfigured server. Second, Flask's routing order: rules without variables are matched before rules with variables. Files that have their own route, such as /robots.txt or /llms.txt, are therefore not affected by this catch-all rule. It is still a good idea to lock that behaviour in with a test.

Batch submission in a single request

A simple GET request works for a single address, but when there are several, a POST request with a JSON body is the better fit. The body carries the host name (host), the key (key), the full address of the key file (keyLocation) and the list of addresses (urlList). The protocol allows at most 10,000 addresses per request. keyLocation is optional when the file sits at the root, but writing it out removes any room for misunderstanding. Every address in the list must belong to the host named in the request.

Our first script read the live sitemap and sent every address in it. Here is a simplified version with error handling added:

import json
import os
import re
import sys
import urllib.error
import urllib.request

SITE = (os.environ.get("SITE_URL") or "https://example.com").rstrip("/")
KEY = (os.environ.get("INDEXNOW_KEY") or "").strip()


def bildir(adresler):
    govde = json.dumps({
        "host": SITE.split("//", 1)[1],
        "key": KEY,
        "keyLocation": f"{SITE}/{KEY}.txt",
        "urlList": adresler[:10000],
    }).encode("utf-8")
    istek = urllib.request.Request(
        "https://api.indexnow.org/indexnow", data=govde,
        headers={"Content-Type": "application/json; charset=utf-8"})
    try:
        with urllib.request.urlopen(istek, timeout=60) as yanit:
            return yanit.status, ""
    except urllib.error.HTTPError as e:
        return e.code, e.read()[:200].decode("utf-8", "replace")


if __name__ == "__main__":
    if not KEY:
        sys.exit("INDEXNOW_KEY is empty")
    adresler = sys.argv[1:]
    if not adresler:
        xml = urllib.request.urlopen(SITE + "/sitemap.xml", timeout=60).read().decode("utf-8")
        adresler = re.findall(r"<loc>([^<]+)</loc>", xml)
    kod, ayrinti = bildir(adresler)
    print("IndexNow", kod, len(adresler), "addresses", ayrinti)

If addresses are given on the command line, the script sends only those; otherwise it sends the addresses in the sitemap. Since urllib is part of the standard library, no extra dependency is needed, which makes it easy to run as a scheduled task on the server.

Reading the response codes correctly

The response codes defined in the IndexNow documentation tell you fairly clearly where a problem lies:

• 200: the submission was received.
• 202: the submission was accepted, but key validation is still pending. Common on the first submission; not an error.
• 400: the request format is invalid; usually broken JSON or a missing field.
• 403: the key is not valid; the key file was not found, or the key inside it does not match the one in the request.
• 422: the addresses do not belong to the host in the request, or the key is not in the expected format.
• 429: too many requests; the engine sees this as potential spam.

The typical causes of a 403 are configuration problems: a key pasted into the environment variable with a leading or trailing space (which is why the code above calls strip()), or a site whose www and non-www addresses behave differently. So the first step is to open the key file in a browser at exactly the keyLocation you send and check its content by eye. The host detail matters too: the addresses you report should be on your canonical host. Make sure your site redirects to a single address; otherwise you are publishing two copies of the same content, both for IndexNow and for search results.

When to submit, and when not to

The purpose of the protocol is to announce change, not to say "my site is still here". Sending the whole sitemap on first setup is reasonable, but after that only addresses that are genuinely new, changed or removed should be sent. We learned this the hard way on one of our own projects: a regularly scheduled task kept re-submitting every address in the sitemap even when no content had changed. That created a pointless load that gave the engines no new information, and it raised the risk of a 429 response as well. The fix was to store the state of the last submission and send only the difference.

If the lastmod values in your sitemap are accurate, finding the difference is easy: keep a file recording which address was sent with which date last time, and compare it with the new sitemap.

import json
import re
import urllib.request
from pathlib import Path

DURUM = Path("indexnow_state.json")


def sitemap_oku(adres):
    xml = urllib.request.urlopen(adres, timeout=60).read().decode("utf-8")
    return dict(re.findall(r"<loc>([^<]+)</loc>\s*<lastmod>([^<]+)</lastmod>", xml))


def degisenler(simdiki):
    onceki = json.loads(DURUM.read_text("utf-8")) if DURUM.exists() else {}
    return [adres for adres, tarih in simdiki.items() if onceki.get(adres) != tarih]


def durumu_kaydet(simdiki):
    DURUM.write_text(json.dumps(simdiki, ensure_ascii=False, indent=1), "utf-8")

The flow is: read the sitemap, find what changed, send it with bildir() if the list is not empty, and save the state only if the response was 200 or 202. A failed submission is then retried on the next run. If you also want to report deleted pages, add the addresses that were in the previous state but are missing from the new sitemap; when the engine visits them it will see a 404 or 410.

This approach has one prerequisite: lastmod must really show the day the content changed. A sitemap that prints today's date on every request makes this difference meaningless and teaches search engines not to trust the field. On this site, the lastmod of a product page comes from that product's latest update entry, and an article's value comes from the article's own date.

What to do for Google

Google does not take part in IndexNow; IndexNow submissions do not reach Google. For Google the route is still the sitemap and Search Console. Google's old sitemap "ping" endpoint was also deprecated in 2023; it is now enough to add the sitemap to Search Console once and point to it with a Sitemap: line in robots.txt. Google states that it uses the lastmod value only as long as it is consistently accurate, so the honest-date rule above applies to Google as well.

When you want one important page crawled quickly, you can request indexing from the URL Inspection tool in Search Console; that is a manual, limited operation and no substitute for batch notification. In practice our setup is: for Bing, Yandex and the other participants we report changed addresses immediately with IndexNow; for Google we keep a sitemap with accurate lastmod values and read the coverage reports in Search Console regularly. We described how the sitemap of a multilingual site is generated in our multilingual site article.

Conclusion

Setting up IndexNow takes a few lines: a key read from an environment variable, a route that publishes it and a small script that sends addresses in one request. The real difference comes from discipline. Report only addresses that changed, read the response codes and retry failed submissions, keep the dates in your sitemap honest, and remember that Google sits outside this protocol. We use the same approach on the static content pages of our own products, such as Pati ve Ben and StockFortuna; in each case the goal is the same: give search engines news of real change, not noise.