Making a Streamlit app visible to search engines: static pages and an ASGI layer

Streamlit is one of the fastest ways to build a data application in Python. A few hundred lines give you charts, tables, forms and filters, and you never have to write front-end JavaScript. We use Streamlit in products such as StockFortuna and VedikRehber. But opening an app to visitors is not the same as opening it to search engines. The way Streamlit works creates a few problems that are easy to miss at first and that directly affect search visibility.

In this article we describe those problems and the two fixes we apply in our own products: generating static content pages that search engines can read, and placing a small ASGI middleware layer in front of the Streamlit server. The code samples are deliberately plain. They do not depend on Starlette or any other framework and will run on any server that speaks the ASGI interface.

The problem: search engines see an empty shell

If you look at the raw HTML of a Streamlit page, you will not find the content. For every address the server returns the same small shell: a few meta tags, a JavaScript bundle and a note saying that JavaScript is required to run the app. The real content is drawn piece by piece after the browser loads the bundle, opens a WebSocket connection to the server and your Python script runs.

This has three consequences for search visibility:

• The content is not in the raw HTML. Google's rendering service can execute JavaScript, but because the content arrives over a WebSocket session and depends on how long the script takes, we found in our own tests that the outcome is not reliable. Other search engines and AI crawlers mostly read only the raw HTML.
• Every address carries the same title. There is one shell, so there is one <title>, one description and one canonical address. In a multipage app every page appears in search results with the same title, or is treated as a copy of the others.
• Addresses that do not exist return 200. Following single-page app logic, an unknown address also receives the shell. For a search engine this is a "soft 404": mistyped or outdated links pile up in the index as if they were real pages.

We also hit a side effect on a live system: the address of a folder on disk (/sozluk, our glossary) was redirected by one layer that appended a trailing slash and by another layer that removed it. The browser ended up with an infinite redirect error. We could only find it by measuring addresses one by one before a user noticed.

First fix: generate static content pages

Most of the content you want search engines to see does not actually need interaction. A summary analysis of a stock, a glossary entry for a technical analysis term or a weekly horoscope reading is text that is read when the page opens. Instead of having Streamlit draw that content, we generate plain HTML files from the same data. The app still runs in Streamlit; the static pages act as entry points and link readers to the relevant interactive screen.

The generator is a simple loop: it reads the data, writes one HTML file per record and collects sitemap lines at the same time. What matters is that each file carries its own title, description, canonical address and language alternates.

import html
from pathlib import Path

SITE = "https://example.com"

TERMS = [
    {"slug": "rsi", "tr": ("RSI nedir?", "Göreli güç endeksinin hesabı ve yorumu."),
     "en": ("What is RSI?", "How the relative strength index is calculated and read.")},
]

def page(lang, title, description, path, alternates, body):
    e = html.escape
    links = "\n".join(
        f'<link rel="alternate" hreflang="{l}" href="{SITE}{p}">' for l, p in alternates)
    return f"""<!doctype html>
<html lang="{lang}">
<head><meta charset="utf-8">
<title>{e(title)}</title>
<meta name="description" content="{e(description)}">
<link rel="canonical" href="{SITE}{path}">
{links}
</head><body><h1>{e(title)}</h1>{body}</body></html>"""

urls = []
for t in TERMS:
    alternates = [("tr", f"/sozluk/{t['slug']}/"), ("en", f"/en/sozluk/{t['slug']}/")]
    for lang, path in alternates:
        title, description = t[lang]
        target = Path("static") / path.strip("/") / "index.html"
        target.parent.mkdir(parents=True, exist_ok=True)
        body = f"<p>{html.escape(description)}</p>"
        target.write_text(page(lang, title, description, path, alternates, body), encoding="utf-8")
        urls.append(f"{SITE}{path}")

Path("static/sitemap-glossary.xml").write_text(
    '<?xml version="1.0" encoding="UTF-8"?>\n'
    '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n'
    + "".join(f"  <url><loc>{html.escape(u)}</loc></url>\n" for u in urls)
    + "</urlset>\n", encoding="utf-8")

A few practical rules come with this approach. First, a static page has to carry real information; thin pages that only say "open the app" are treated by search engines as low-value content. Second, every language version should live at its own address, and hreflang should point only to translations that actually exist. Third, the pages must be regenerated whenever the data changes; tying this to a nightly job keeps it from being forgotten. Streamlit's own static file serving publishes files only under /app/static/; to serve pages at root addresses such as /sozluk/ you need either a reverse proxy in front or a start-up wrapper around a static file handler that serves the root folder. The layer below is part of that second route.

Second fix: an ASGI layer in front of the server

Static pages solve the content problem, but address behaviour, duplicate titles and soft 404s still live in the app itself. Fixing them needs a layer that sees each request before it reaches Streamlit. Because the Starlette-based server in recent Streamlit versions uses the ASGI interface, that layer can be written as an ordinary ASGI middleware.

In ASGI an application is an asynchronous callable that takes scope, receive and send. A middleware is a wrapper with the same signature: it inspects the request, answers it itself if it wants to, and otherwise passes it to the inner application. The first example adds security headers to every HTTP response if they are missing. It does not touch WebSocket or lifespan messages, which matters so that Streamlit's WebSocket connection keeps working.

SECURITY_HEADERS = (
    (b"x-content-type-options", b"nosniff"),
    (b"referrer-policy", b"strict-origin-when-cross-origin"),
    (b"x-frame-options", b"SAMEORIGIN"),
)

class SecurityHeaders:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope, receive, send):
        if scope["type"] != "http":
            await self.app(scope, receive, send)
            return

        async def send_wrapper(message):
            if message["type"] == "http.response.start":
                headers = list(message.get("headers", []))
                present = {name.lower() for name, _ in headers}
                headers += [(n, v) for n, v in SECURITY_HEADERS if n not in present]
                message = dict(message, headers=headers)
            await send(message)

        await self.app(scope, receive, send_wrapper)

A real 404 for unknown addresses

The second middleware keeps a list of routes the app knows. If a request is not on that list, does not match a file on disk and is not one of Streamlit's own internal endpoints, it returns a plain 404 page. Instead of maintaining the route list by hand you can read it from the app file; we parse the page definitions from the source code, so the list updates itself when a new page is added. The example below uses a fixed set for simplicity.

import os

KNOWN_ROUTES = {"", "markets", "stock", "contact"}
PROTECTED_PREFIXES = ("_stcore", "static", "media", "component")
PAGE_404 = ("<!doctype html><html lang=\"en\"><meta charset=\"utf-8\">"
            "<title>Page not found</title><h1>404</h1>"
            "<p><a href=\"/\">Back to the home page</a></p>").encode("utf-8")

def exists_on_disk(root, path):
    parts = [p for p in path.split("/") if p]
    if any(p in (".", "..") or "\\" in p for p in parts):
        return False
    full = os.path.join(root, *parts)
    return os.path.isfile(full) or os.path.isfile(os.path.join(full, "index.html"))

class RealNotFound:
    def __init__(self, app, root):
        self.app = app
        self.root = root

    async def __call__(self, scope, receive, send):
        if scope["type"] == "http" and scope["method"] in ("GET", "HEAD"):
            route = scope["path"].strip("/")
            known = route in KNOWN_ROUTES or route.split("/")[0] in PROTECTED_PREFIXES
            if not known and not exists_on_disk(self.root, scope["path"]):
                await send({"type": "http.response.start", "status": 404, "headers": [
                    (b"content-type", b"text/html; charset=utf-8"),
                    (b"content-length", str(len(PAGE_404)).encode()),
                    (b"cache-control", b"no-cache"),
                    (b"x-robots-tag", b"noindex"),
                ]})
                await send({"type": "http.response.body", "body": PAGE_404})
                return
        await self.app(scope, receive, send)

The .. check in exists_on_disk may look unnecessary, but it is mandatory wherever you turn an incoming path into a file system path. The protected prefix list is just as critical: if you accidentally send Streamlit's WebSocket and health endpoints to a 404, the app will not open at all. After every upgrade you need to confirm that this list is still correct.

Folder addresses and per-route metadata

Because we generate static pages as index.html inside folders, we set two rules: a request for /sozluk/ is rewritten to the folder's index.html and returns 200, and a request for /sozluk is sent to /sozluk/ with a 301. We assume the inner app wrapped by this layer is a static file app serving the root folder at the site root (in our case the static handler wrapped at start-up; elsewhere it could be a reverse proxy or a Starlette StaticFiles mount). It is important to write the redirect target as a relative path rather than an absolute URL: behind a reverse proxy the app sees the request as HTTP, and if it builds an absolute address it can drop the visitor from HTTPS to HTTP.

import os
from urllib.parse import quote

class FolderAddress:
    def __init__(self, app, root):
        self.app = app
        self.root = root

    async def __call__(self, scope, receive, send):
        if scope["type"] == "http" and scope["method"] in ("GET", "HEAD"):
            path = scope["path"]
            parts = [p for p in path.split("/") if p]
            safe = parts and not any(p in (".", "..") or "\\" in p for p in parts)
            if safe and os.path.isfile(os.path.join(self.root, *parts, "index.html")):
                if path.endswith("/"):
                    new = path + "index.html"
                    scope = dict(scope, path=new, raw_path=quote(new).encode("ascii"))
                else:
                    target = quote(path) + "/"
                    query = scope.get("query_string", b"")
                    if query:
                        target += "?" + query.decode("latin-1")
                    await send({"type": "http.response.start", "status": 301,
                                "headers": [(b"location", target.encode("latin-1")),
                                            (b"content-length", b"0")]})
                    await send({"type": "http.response.body", "body": b""})
                    return
        await self.app(scope, receive, send)

# Order: the leftmost wrapper sees the request first
# app = SecurityHeaders(RealNotFound(FolderAddress(asgi_app, ROOT), ROOT))

For per-route metadata we extend the same layer: when a request hits a known Streamlit route, we read the shell HTML and write that route's title, description, canonical address and social sharing tags on the server. As a result /markets and /contact no longer share a title. The replacement is plain string processing; the values must always be escaped with html.escape:

import html
import re

def write_metadata(shell: str, title: str, description: str, canonical: str) -> str:
    e = html.escape
    head = (f"<title>{e(title)}</title>"
            f'<meta name="description" content="{e(description)}">'
            f'<link rel="canonical" href="{e(canonical)}">')
    shell = re.sub(r"<title>.*?</title>", "", shell, count=1, flags=re.S)
    return shell.replace("</head>", head + "</head>", 1)

Wiring the layer into Streamlit: a version warning

Everything so far is standard ASGI. The hard part is placing the layer inside Streamlit. Streamlit builds its own server and does not offer an official hook for adding middleware. On Streamlit's Starlette-based server (tested with version 1.58) we use a small start-up script that wraps the internal functions which build the static file handler and the middleware list, before the app is created. It works, but because it relies on internal modules it is not a public API; module names and function signatures can change in the next release.

We manage that risk in three ways. In the start-up script every patch sits in its own try/except block; if one fails the app still starts and the log records which step was skipped. We pin the Streamlit version in the dependency file. And after every upgrade we run a test that measures the same list of addresses: home page 200, unknown address 404, folder address 301 followed by 200, and a different title on every route. If you want a less fragile route, you can implement the same rules in a reverse proxy in front of the app, such as Nginx, Caddy or Traefik. Then you never touch Streamlit's internals, but you have to keep the route list in two places.

Caching, robots and verification

On sites behind a content delivery network we ran into one more small but annoying problem: root files such as robots.txt, sitemaps and llms.txt were cached for a very long time under static file defaults. When a sitemap changed, search engines kept seeing the old version. Setting Cache-Control: no-cache or a short lifetime for these files was enough.

Verification means looking at the raw HTTP response, not at the browser. We check status codes and headers with curl -I and each route's <title> with curl -s. The URL inspection tool in Search Console also shows how Google sees the page, but since changes can take days to reach the index, local and live measurement should always come first.

Conclusion

You can offer search engines a readable site without giving up Streamlit. Moving readable content into static pages solves most of the problem; address behaviour, per-route titles, real 404s and security headers are fixed with small, framework-independent ASGI middleware. The one thing to keep in mind is that wiring this layer into Streamlit relies on its internals, so every upgrade should be measured. You can see the products where we use this approach on the StockFortuna and VedikRehber pages.