If you're deploying a Rails app on Heroku, you've probably hit that awkward moment where static SEO files like sitemap.xml and robots.txt become a pain. The traditional sitemap_generator gem wants to write to your filesystem or S3, but Heroku's ephemeral filesystem makes that annoying. Here's how to build dynamic endpoints that generate these files on-the-fly with smart caching, no external dependencies needed.

Why Dynamic Endpoints?

Static file generation works great on traditional servers, but Heroku's dyno filesystem resets on every deploy. You could setup S3 buckets and configure sitemap_generator to upload there, or you could just... not. Dynamic endpoints give you flexibility to include real-time data (like recently updated blog posts or products) without running cron jobs or deploy hooks. Plus, you control caching exactly how you want.

Setting Up Routes and Controller Actions

First, add routes that force the correct content types:

# config/routes.rb
Rails.application.routes.draw do
  get "robots", to: "seo#robots", defaults: { format: :text }
  get "sitemap", to: "seo#sitemap", defaults: { format: :xml }
end

Then create a controller with methods that leverage Rails.cache to avoid regenerating these files on every request:

# app/controllers/seo_controller.rb
class SeoController < ApplicationController
  def robots
    respond_to do |format|
      format.text do
        render plain: generate_robots_txt, content_type: "text/plain"
      end
    end
  end

  def sitemap
    respond_to do |format|
      format.xml do
        render xml: generate_sitemap_xml, content_type: "application/xml"
      end
    end
  end

  private

  def generate_robots_txt
    Rails.cache.fetch("robots_txt", expires_in: 1.day) do
      <<~ROBOTS
        User-agent: *
        Allow: /

        Sitemap: #{request.base_url}/sitemap.xml

        # Popular pages
        #{static_page_urls.join("\n")}
      ROBOTS
    end
  end

  def generate_sitemap_xml
    Rails.cache.fetch("sitemap_xml", expires_in: 1.day) do
      builder = Nokogiri::XML::Builder.new(encoding: "UTF-8") do |xml|
        xml.urlset(xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9") do
          static_pages_data.each do |page|
            xml.url do
              xml.loc page[:url]
              xml.changefreq page[:changefreq]
              xml.priority page[:priority]
              xml.lastmod Time.current.iso8601
            end
          end
        end
      end
      builder.to_xml
    end
  end

  def static_page_urls
    [root_url, about_url, contact_url]
  end

  def static_pages_data
    [
      { url: root_url, changefreq: "weekly", priority: 1.0 },
      { url: about_url, changefreq: "monthly", priority: 0.5 },
      { url: contact_url, changefreq: "monthly", priority: 0.5 }
    ]
  end
end

Performance Considerations: The Big Problem

Here's where things get spicy. If you naively try to include 100k+ database records in your sitemap, you'll murder your dyno. Loading massive result sets into memory is a one-way ticket to timeout city. A few strategies to avoid this:

1. Batch Processing with findinbatches

Never use Product.all.each - use find_in_batches to load records in chunks:

def generate_sitemap_xml
  Rails.cache.fetch("sitemap_xml", expires_in: 1.day) do
    builder = Nokogiri::XML::Builder.new(encoding: "UTF-8") do |xml|
      xml.urlset(xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9") do
        # Static pages first
        add_static_pages_to_sitemap(xml)

        # Then batch process products
        Product.find_in_batches(batch_size: 1000) do |batch|
          batch.each do |product|
            xml.url do
              xml.loc product_url(product)
              xml.lastmod product.updated_at.iso8601
              xml.changefreq "daily"
              xml.priority 0.8
            end
          end
        end
      end
    end
    builder.to_xml
  end
end

2. Limit Records Strategically

Do you really need every single record? Consider limiting to recently updated items or a representative sample:

# Only include products updated in last 90 days
Product.where("updated_at > ?", 90.days.ago)
  .find_in_batches(batch_size: 1000) do |batch|
  # build sitemap entries
end

# Or limit to a maximum count
Product.order(updated_at: :desc)
  .limit(10_000)
  .find_in_batches(batch_size: 1000) do |batch|
  # build sitemap entries
end

3. Use Sitemap Indexes for Large Sites

If you've got tons of content, implement a sitemap index that points to multiple sitemap files (paginated by category, date, etc):

# GET /sitemap.xml returns index
# GET /sitemap/products.xml returns product sitemap
# GET /sitemap/blog.xml returns blog sitemap

Other Use Cases for Dynamic SEO Endpoints

This pattern isn't just for sitemaps. You can use it for:

  • Dynamic RSS/Atom feeds - Generate feeds based on user preferences or content categories
  • Dynamic manifest.json for PWAs - Customize app manifests based on subdomain or user settings
  • robots.txt variations - Different crawl rules for staging vs production environments
  • OpenGraph/Twitter Card meta tags - Generate social sharing metadata dynamically
  • Structured data/Schema.org JSON-LD - Build SEO-rich structured data from your database

Dependencies You'll Need

For the XML generation approach shown above, add Nokogiri to your Gemfile:

# Gemfile
gem 'nokogiri'

Then run:

bundle install

Nokogiri is usually already in your Rails app for other purposes, so you might already have it. Rails.cache works out of the box with Rails - just make sure you're using a proper cache store in production (not :memory_store). On Heroku, Memcachier or Redis are solid choices:

# config/environments/production.rb
config.cache_store = :mem_cache_store, ENV['MEMCACHIER_SERVERS'],
  { username: ENV['MEMCACHIER_USERNAME'],
    password: ENV['MEMCACHIER_PASSWORD'] }

Or with Redis:

# Add to Gemfile
gem 'redis'

# config/environments/production.rb
config.cache_store = :redis_cache_store, { url: ENV['REDIS_URL'] }

That's it! You've got SEO-friendly, Heroku-compatible dynamic endpoints that won't choke your app when search bots come knocking.