# Dynamic Sitemap and Robots.txt in Rails: Ditching Static Files for Heroku-Friendly Endpoints

If you're deploying a Rails app on Heroku, you've probably hit that awkward moment where static SEO files like `sitemap.xml` and `robots.txt` become a pain. The traditional `sitemap_generator` gem wants to write to your filesystem or S3, but Heroku's ephemeral filesystem makes that annoying. Here's how to build dynamic endpoints that generate these files on-the-fly with smart caching, no external dependencies needed.

## Why Dynamic Endpoints?

Static file generation works great on traditional servers, but Heroku's dyno filesystem resets on every deploy. You could setup S3 buckets and configure `sitemap_generator` to upload there, or you could just... not. Dynamic endpoints give you flexibility to include real-time data (like recently updated blog posts or products) without running cron jobs or deploy hooks. Plus, you control caching exactly how you want.

## Setting Up Routes and Controller Actions

First, add routes that force the correct content types:

```ruby
# config/routes.rb
Rails.application.routes.draw do
  get "robots", to: "seo#robots", defaults: { format: :text }
  get "sitemap", to: "seo#sitemap", defaults: { format: :xml }
end
```

Then create a controller with methods that leverage Rails.cache to avoid regenerating these files on every request:

```ruby
# app/controllers/seo_controller.rb
class SeoController < ApplicationController
  def robots
    respond_to do |format|
      format.text do
        render plain: generate_robots_txt, content_type: "text/plain"
      end
    end
  end

  def sitemap
    respond_to do |format|
      format.xml do
        render xml: generate_sitemap_xml, content_type: "application/xml"
      end
    end
  end

  private

  def generate_robots_txt
    Rails.cache.fetch("robots_txt", expires_in: 1.day) do
      <<~ROBOTS
        User-agent: *
        Allow: /

        Sitemap: #{request.base_url}/sitemap.xml

        # Popular pages
        #{static_page_urls.join("\n")}
      ROBOTS
    end
  end

  def generate_sitemap_xml
    Rails.cache.fetch("sitemap_xml", expires_in: 1.day) do
      builder = Nokogiri::XML::Builder.new(encoding: "UTF-8") do |xml|
        xml.urlset(xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9") do
          static_pages_data.each do |page|
            xml.url do
              xml.loc page[:url]
              xml.changefreq page[:changefreq]
              xml.priority page[:priority]
              xml.lastmod Time.current.iso8601
            end
          end
        end
      end
      builder.to_xml
    end
  end

  def static_page_urls
    [root_url, about_url, contact_url]
  end

  def static_pages_data
    [
      { url: root_url, changefreq: "weekly", priority: 1.0 },
      { url: about_url, changefreq: "monthly", priority: 0.5 },
      { url: contact_url, changefreq: "monthly", priority: 0.5 }
    ]
  end
end
```

## Performance Considerations: The Big Problem

Here's where things get spicy. If you naively try to include 100k+ database records in your sitemap, you'll murder your dyno. Loading massive result sets into memory is a one-way ticket to timeout city. A few strategies to avoid this:

**1. Batch Processing with find_in_batches**

Never use `Product.all.each` - use `find_in_batches` to load records in chunks:

```ruby
def generate_sitemap_xml
  Rails.cache.fetch("sitemap_xml", expires_in: 1.day) do
    builder = Nokogiri::XML::Builder.new(encoding: "UTF-8") do |xml|
      xml.urlset(xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9") do
        # Static pages first
        add_static_pages_to_sitemap(xml)

        # Then batch process products
        Product.find_in_batches(batch_size: 1000) do |batch|
          batch.each do |product|
            xml.url do
              xml.loc product_url(product)
              xml.lastmod product.updated_at.iso8601
              xml.changefreq "daily"
              xml.priority 0.8
            end
          end
        end
      end
    end
    builder.to_xml
  end
end
```

**2. Limit Records Strategically**

Do you really need every single record? Consider limiting to recently updated items or a representative sample:

```ruby
# Only include products updated in last 90 days
Product.where("updated_at > ?", 90.days.ago)
  .find_in_batches(batch_size: 1000) do |batch|
  # build sitemap entries
end

# Or limit to a maximum count
Product.order(updated_at: :desc)
  .limit(10_000)
  .find_in_batches(batch_size: 1000) do |batch|
  # build sitemap entries
end
```

**3. Use Sitemap Indexes for Large Sites**

If you've got tons of content, implement a sitemap index that points to multiple sitemap files (paginated by category, date, etc):

```ruby
# GET /sitemap.xml returns index
# GET /sitemap/products.xml returns product sitemap
# GET /sitemap/blog.xml returns blog sitemap
```

## Other Use Cases for Dynamic SEO Endpoints

This pattern isn't just for sitemaps. You can use it for:

- **Dynamic RSS/Atom feeds** - Generate feeds based on user preferences or content categories
- **Dynamic manifest.json for PWAs** - Customize app manifests based on subdomain or user settings
- **robots.txt variations** - Different crawl rules for staging vs production environments
- **OpenGraph/Twitter Card meta tags** - Generate social sharing metadata dynamically
- **Structured data/Schema.org JSON-LD** - Build SEO-rich structured data from your database

## Dependencies You'll Need

For the XML generation approach shown above, add Nokogiri to your Gemfile:

```ruby
# Gemfile
gem 'nokogiri'
```

Then run:

```bash
bundle install
```

Nokogiri is usually already in your Rails app for other purposes, so you might already have it. Rails.cache works out of the box with Rails - just make sure you're using a proper cache store in production (not `:memory_store`). On Heroku, Memcachier or Redis are solid choices:

```ruby
# config/environments/production.rb
config.cache_store = :mem_cache_store, ENV['MEMCACHIER_SERVERS'],
  { username: ENV['MEMCACHIER_USERNAME'],
    password: ENV['MEMCACHIER_PASSWORD'] }
```

Or with Redis:

```ruby
# Add to Gemfile
gem 'redis'

# config/environments/production.rb
config.cache_store = :redis_cache_store, { url: ENV['REDIS_URL'] }
```

That's it! You've got SEO-friendly, Heroku-compatible dynamic endpoints that won't choke your app when search bots come knocking.