In the last post we looked at how a request gets from the internet to the front door of Django, the wsgi.py file. WSGI, which is short for Web Server Gateway Interface, is the topic of today’s post, as it lays the groundwork for understanding how Django handles requests. We’ll cover the key components of WSGI, and then take a look at how these are implemented in Django. This isn’t a comprehensive explanation of WSGI, in part because Django doesn’t use some of the optional pieces of the interface. If you’re interested in understanding it more fully (and have a couple hours to spare), the official proposal document is an interesting read.

What problem WSGI solves

According to the official spec, WSGI is “a proposed standard interface between web servers and Python web applications or frameworks, to promote web application portability across a variety of web servers.”

This allows application developers to pick a Python framework without worrying about underlying infrastructure, and vice versa for infrastructure engineers. Application developers don’t have to know how it works, or even that it exists, so long as the framework and server are compatible.

WSGI Middleware

Thus far, we’ve assumed that the server described is the web server, and that the application is Django or a similar web framework. This actually isn’t necessary. Since WSGI is an interface, its job is to define how two programs interact with one another, not what type of programs they are.

This allows for something called middleware-chaining. Middleware is a program that sits between the server and Django, and adds additional functionality based on the request or the response.

Simple Case:

Single Middleware Instance:

Middleware Chaining:

As long as the middleware fulfills the WSGI contract, then it can happily serve as the application from the server’s perspective, and the server from Django’s perspective. Chaining is possible by putting multiple different middleware instances in a row between the server and Django. No part of this path is aware of any of the others; it just knows it’s interfacing with a WSGI-compatible program.

How the server first accesses Django

Per the WSGI spec, the server is expecting to access the framework and get back some type of callable  (named application in the spec) This callable accepts two arguments from the server, a dictionary of environment variables (environ) and another callable (start_response). Django will then use the data in environ to complete the request, and will use start_response to pass the response back to the server.

Let’s recap:

1. Server calls a function on the application, and gets back a callable.
2. Server uses that callable to pass in a dictionary of data, environ,  and a callable of its own, start_response, to Django.
3. Django uses environ data to complete the request and get the response.
4. Django uses start_response to send response data back to the server.

If we look at how this is implemented within Django, we’ll see that it starts with the argument passed to the server (in this case Gunicorn) when it is first started up. As an example I’m using the code from a project of mine called locode.

gunicorn locode.wsgi

This is the dotpath of the wsgi.py file where Gunicorn will look to find the application callable. Here we’ll find the following code:

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "locode.settings")

application = get_wsgi_application()

The last line is defining the application callable, which Gunicorn now has access to.

Following the dependency tree up to django.core.wsgi the following code can be found (I’ve excluded comments for readability).

import django

from django.core.handlers.wsgi import WSGIHandler

def get_wsgi_application():

    django.setup(set_prefix=False)

    return WSGIHandler()

The first line of get_wsgi_application sets up Django so that it’s various components are ready to receive request data. For the time being we won’t worry about that, as it will be the topic for a future post.

The second line, return WSGIHandler() initializes the actual WSGI callable (i.e. the application variable). Up the tree (in yet another wsgi.py file) we can see the actual code laid out, the function signatures of which are below.

class WSGIHandler(base.BaseHandler):

request_class = WSGIRequest

def __init__(self, *args, **kwargs):

def __call__(self, environ, start_response):

We can see how the WSGI callable is created and returned to the server. This architecture allows the WSGIHandler class to be instantiated as an object (thus possessing state), yet to be called like function, receiving the environ data and start_response callable, which fulfills step 1 of the WSGI requirements listed above.

Passing the request to Django

The previous step occurs when the server is first started, after which it waits for requests to come in through the appropriate ports. When an HTTP request is received by the server, that’s the time for the server to make use of the application callable.

The environ dictionary, the first argument the server passes into the application callable,  can be a bit of a black box, as it’s used to pass in multiple different pieces of data, which vary depending on the specific server and configuration.  Included in this list are HTTP/request data, (method, query string, content length, content type, port number, headers, etc.), operating system environment variables (such as user defined private keys), WSGI variables (version number, input and error streams, process and thread data), and additional server variables.

In general, all the data that Django needs to receive from the server comes through the environ dictionary.  This, along with the start_response callable, are the two required values that the server passes to Django. We’ll return to start_response in a moment when it’s time to return the response to the server, but now Django has everything it needs to process the request, meaning we’ve completed step 2.

Turning a request into a response

While this may seem like the most important step, from WSGI’s perspective how this happens doesn’t matter. This is a standard part of developing any interface. By determining the rules of how two systems interact, but not how either one of them accomplishes their part of the contract, both systems have fewer dependencies and are free to make changes to their internal workings as long as they keep up their sides of the contract.

The WSGIHandler.__call__() function listed above (aka the application callable) also doesn’t care how the rest of Django handles the request. This is because its job is merely to facilitate Django’s side of WSGI, and it hands off request processing to other specialized functions. This is why it makes sense for the __call__() function to have only the following lines of code for request handling:

request = self.request_class(environ)

response = self.get_response(request)

In the first line, a designated function parses the environ dictionary and separates out the HTTP request-specific data. The second line takes in the request object, and hands it off to get_response to do the work, receiving a response object that’s ready to be returned to the server.

We now have our response. More specifically, the Django-defined application callable function that the server has called has it’s response data. This isn’t actually an HTTP Response as far as the internet is concerned. It’s just a response object. The server handles the actual HTTP side of things, and then sends and receives data that’s formatted in a standard Python manner. Django doesn’t ever interact with the true HTTP request, just an abstraction of it.

We’ll cover the inner workings of get_response in a future post, but for now step 3 is complete.

Returning the response to the server

Now that we have our response object, it’s time for Django to use thestart_response callable (passed into the application callable by the server) to begin passing the response back.

This primarily consists of two steps:

1. Invoking start_response with its two required arguments, status (e.g. “200 OK”) and headers, which is list of header tuples.
2. Returning an iterator containing the response body.

This might seem like a strange way of returning data to the server. Why doesn’t  start_response just take a third required argument called body, thus simplifying the whole process?

For many cases, such as returning a simple web page, that would work. In fact, for simple responses, the iterator returned to the server is just a one-element list. But for larger amounts of data, or streams of data, this isn’t feasible. The server needs a way of getting the data in smaller blocks, processing the block in whatever manner is necessary, and then asking Django for the next block.

The iterator is what makes this possible. The most basic form of iterator is a list, which you’re probably familiar with:

for i in [1,2,3]: # Iterating over a list.
    print(i)

Under the hood, an iterator is just a class that has __iter__ and __next__ methods. With a list the __next__ method returns the next local value, which is already part of the list object’s state. But there’s no rule saying that the data returned by __next__ has to already be a property of the object instance. The only requirement is that a block of data get returned.

This flexibility is what makes the iterator such a useful tool for Django to pass back to the server.  If the response is simple, the iterator need only be a one-element list. But if there is more data, the iterator can be a more complex object that implements __next__ by pulling data from the buffer that Django placed it in. From the server’s perspective, it makes no difference.

Now that the response has been returned to the server, Django doesn’t need to worry about it. The request has successfully been processed and a response has been returned. All Django has to do is wait for the next request to come in so the process can start all over again.