Anton Kopylov

index_now: Telling Search Engines What Changed

rubyrailsseoopen-source

When a page changes, search engines find out the next time they crawl it, and that can take days. IndexNow is the shortcut: you POST the changed URLs to api.indexnow.org, and it passes them on to Bing (which also feeds ChatGPT search and Copilot), Yandex, Seznam, Naver and Yep. Google is not part of it, so none of this touches Google.

The protocol is one HTTP request. The hard part is deciding what to send. My first version lived inside the Reverse Image Search Anywhere backend, tied to its page list, a cron job on the host and a single site-wide watermark. I wanted the same thing in every Rails app I run, so it is a gem now, index_now, and ExtensionDash is the first app on it.

gem "index_now"

bin/rails generate index_now:install
bin/rails db:migrate

The sitemap is the only signal

The gem does not watch models or hash pages. It reads the app’s own sitemap and announces every URL whose <lastmod> is newer than the one it last announced for that URL. Each URL it sends gets a row with that lastmod, so a second run with nothing changed sends nothing. That makes it safe to call as often as you like, which matters: engines learn to ignore hosts that keep announcing pages that did not change.

So the work moves to where it belongs, the <lastmod>. Two kinds of change feed it:

  • Copy: a date you bump by hand when a page’s text changes. Never the deploy time, or every deploy announces every page.
  • Data: the time the data behind the page last changed. ExtensionDash’s store stats pages use the newest stats row.

Both triggers are the same call. A Kamal post-deploy hook runs bin/rails index_now:announce for copy, and the job that writes the data calls it for data:

class StoreStatsJob < ApplicationJob
  def perform
    StoreStats.measure!
    IndexNow.announce_changes
  end
end

Reading the sitemap without the network

The sitemap is read in-process, through Rails.application.call, not over HTTP. A job does not depend on DNS, the load balancer, or the app being able to reach itself. Two details took a second pass. The request runs on its own thread, because the Rails executor would otherwise reset the calling job’s Current attributes and query cache halfway through. And the host has to pass config.hosts, so a 403 comes back with exactly that hint in the message.

It never raises

A failed announcement must not fail the deploy or the job that asked for it. IndexNow.announce_changes returns a result instead: the URLs it sent, or the error, which is also logged and reported through Rails.error. Nothing is recorded for a URL until the POST that carried it succeeded, so the next call retries it, with no retry code anywhere. Outside production it stays off and only logs what it would have sent.

The rest

  • bin/rails index_now:pending lists what the next announce would send. index_now:announce_all sends everything again, for after a key rotation.
  • The key file goes in public/ and in git. It is not a secret: it proves you control the host and nothing more.
  • Errors from the API come back as sentences. A 403 names the key URL that has to answer, and a 422 says the URLs or the key do not match the host.

It needs Rails 8 and Ruby 3.3 or newer, reads one sitemap per app, and does not support sitemap index files yet. It is MIT licensed and on RubyGems. Issues and pull requests are welcome on GitHub.

← Blog