Generate sitemap and robots.txt in django

In this tutorial, I will show you how to add the sitemap.xml and robots.txt files to your django web application. But what exactly is sitemap and robot files and why even bother creating them.

A sitemap is a file that provides information about the pages, and other files on your website, and helps engines like Google crawl and rank your site more efficiently.

A robot.txt is just a normal text file that lives at /robots.txt on your site and tells search engines bots which URLs they can access.

Start new django project

django-admin startproject project
cd project
python manage.py startapp core

Luckily, django provides a sitemap framework that you can use right out of the box to quickly create sitemap files. Just add the 'django.contrib.sitemaps' to the INSTALLED_APPS settings.

Add to installed app settings

project/settings.py
INSTALLED_APPS = [
    ...
    'django.contrib.sitemaps',
    'core',
    ...
]

Create a new blog model

To show how you can generate a dynamic sitemap let's create a simple blog model with just two fields(title and content).

core/models.py
from django.db import models

class Blog(models.Model):
    title = models.CharField(max_length=50)
    content = models.TextField()

    def __str__(self):
        return self.title

Add blog to admin

Register the model to admin

core/admin.py
from django.contrib import admin
from core.models import Blog
    
admin.site.register(Blog)

run migrations

python manage.py makemigrations
python manage.py migrate

Implement Views

For demonstration, I am creating three views home, about, and a blog detail page. The code is simple and should be self-explanatory.

core/views.py
from django.shortcuts import render, get_object_or_404
from django.http import HttpResponse
from core.models import Blog

def home(request):
    return HttpResponse("Home Page")

def about(request):
    return HttpResponse("About Page")

def blog(request, pk):
    blog = get_object_or_404(Blog, pk=pk)
    response = f"<h1>{blog.title}</h1> <p>{blog.content}</p>"
    return HttpResponse(response)

Add urls

Let's create the urls for our three views.

project/urls.py
from django.contrib import admin
from django.urls import path
from core import views

urlpatterns = [
    path('admin/', admin.site.urls),

    path('', views.home, name='home'),
    path('about/', views.about, name='about'),
    path('blog/<int:pk>/', views.blog, name='blog'),

Create Sitemaps

To create a sitemap create a new sitemap.py file inside the core app(Note: sitemap code can live anywhere inside your codebase).

core/sitemap.py
from django.contrib.sitemaps import Sitemap
from core.models import Blog
from django.urls import reverse


class BlogSitemap(Sitemap):
    changefreq = "never"
    priority = 0.9

    def items(self):
        return Blog.objects.all()
    
    def location(self,obj):
        return reverse('blog', args=[obj.pk])

Just like we create django forms and models, we imported the Sitemap class from the 'django.contrib.sitemaps' and created the 'BlogSitemap' for our blogs.

Here we specified few attributes like 'changefreq' that tell how frequently the content on the page changes and 'priority is the importance of page with respect to other pages. It should be a float between 0.0 and 1.0. You can read about the other attributes and values from the official documentation.

Then we created the items methods that should either return the queryset of any model or any sequence of relevant values.

Finally, we created the optional location(by default django calls the get_absolute_url() method of the model) method that returns the url of the blog.

Now lets create the sitemap for our static pages(home and about).

class StaticSitemap(Sitemap):
    changefreq = "yearly"

    def items(self):
        return ['home', 'about']

    def location(self, item):
        return reverse(item)

Add the sitemap url

project/urls.py
from django.contrib import admin
from django.urls import path
from core import views

from django.contrib.sitemaps.views import sitemap
from core.sitemap import BlogSitemap, StaticSitemap 

sitemaps = {
    'blog': BlogSitemap,
    'static': StaticSitemap
}

urlpatterns = [
    path('admin/', admin.site.urls),

    path('', views.home, name='home'),
    path('about/', views.about, name='about'),
    path('blog/<int:pk>/', views.blog, name='blog'),

    path('sitemap.xml', sitemap, {'sitemaps': sitemaps},
        name='django.contrib.sitemaps.views.sitemap'),
]

Here we first imported all our sitemap classes and django's sitemap view and then we created the dictionary of our sitemaps. and integrated the sitemap at /sitemap.xml inside the urlpatterns list.

Now run the development server and go to the http://localhost:8000/sitemap.xml

python manage.py runserver
Sitemap added to django app

Generate Robots.txt

Creating the robot.txt file is simple all you need to do is add a new URL /robots.txt inside the URL patterns pointing to a template view that returns a text file as an HTTP response.

project/urls.py

# rest of the code is same
# import this at the top
from django.views.generic.base import TemplateView

urlpatterns = [
    # rest of the urls is same. just add this at the end
    path("robots.txt",TemplateView.as_view(
        template_name="core/robots.txt", content_type="text/plain")),
]

Here is the content for the robots.txt file

core/templates/core/robots.txt
User-agent: *

Head over to http://localhost:8000/robots.txt and you should be able to see the following output.

robots.txt added to django app