In the last post we looked at how a request gets from the internet to the front door of Django, the wsgi.py file. WSGI, which is short for Web Server Gateway Interface, is the topic of today’s post, as it lays the groundwork for understanding how Django handles requests. We’ll cover the key components of WSGI, and then take a look at how these are implemented in Django. This isn’t a comprehensive explanation of WSGI, in part because Django doesn’t use some of the optional pieces of the interface. If you’re interested in understanding it more fully (and have a couple hours to spare), the official proposal document is an interesting read.
What problem WSGI solves
According to the official spec, WSGI is “a proposed standard interface between web servers and Python web applications or frameworks, to promote web application portability across a variety of web servers.”
This allows application developers to pick a Python framework without worrying about underlying infrastructure, and vice versa for infrastructure engineers. Application developers don’t have to know how it works, or even that it exists, so long as the framework and server are compatible.
Thus far, we’ve assumed that the server described is the web server, and that the application is Django or a similar web framework. This actually isn’t necessary. Since WSGI is an interface, its job is to define how two programs interact with one another, not what type of programs they are.
This allows for something called
middleware-chaining. Middleware is a program that sits between the server and Django, and adds additional functionality based on the request or the response.
Single Middleware Instance:
As long as the middleware fulfills the WSGI contract, then it can happily serve as the application from the server’s perspective, and the server from Django’s perspective. Chaining is possible by putting multiple different middleware instances in a row between the server and Django. No part of this path is aware of any of the others; it just knows it’s interfacing with a WSGI-compatible program.
How the server first accesses Django
Per the WSGI spec, the server is expecting to access the framework and get back some type of callable (named
application in the spec) This callable accepts two arguments from the server, a dictionary of environment variables (
environ) and another callable (
start_response). Django will then use the data in
environ to complete the request, and will use
start_response to pass the response back to the server.
- Server calls a function on the application, and gets back a callable.
- Server uses that callable to pass in a dictionary of data,
environ, and a callable of its own,
start_response, to Django.
- Django uses
environdata to complete the request and get the response.
- Django uses
start_responseto send response data back to the server.
If we look at how this is implemented within Django, we’ll see that it starts with the argument passed to the server (in this case Gunicorn) when it is first started up. As an example I’m using the code from a project of mine called
This is the dotpath of the
wsgi.py file where Gunicorn will look to find the
application callable. Here we’ll find the following code:
import os from django.core.wsgi import get_wsgi_application os.environ.setdefault("DJANGO_SETTINGS_MODULE", "locode.settings") application = get_wsgi_application()
The last line is defining the
application callable, which Gunicorn now has access to.
Following the dependency tree up to
django.core.wsgi the following code can be found (I’ve excluded comments for readability).
import django from django.core.handlers.wsgi import WSGIHandler def get_wsgi_application(): django.setup(set_prefix=False) return WSGIHandler()
The first line of
get_wsgi_application sets up Django so that it’s various components are ready to receive request data. For the time being we won’t worry about that, as it will be the topic for a future post.
The second line,
return WSGIHandler() initializes the actual WSGI callable (i.e. the
application variable). Up the tree (in yet another
wsgi.py file) we can see the actual code laid out, the function signatures of which are below.
class WSGIHandler(base.BaseHandler): request_class = WSGIRequest def __init__(self, *args, **kwargs): def __call__(self, environ, start_response):
We can see how the WSGI callable is created and returned to the server. This architecture allows the
WSGIHandler class to be instantiated as an object (thus possessing state), yet to be called like function, receiving the
environ data and
start_response callable, which fulfills step 1 of the WSGI requirements listed above.
Passing the request to Django
The previous step occurs when the server is first started, after which it waits for requests to come in through the appropriate ports. When an HTTP request is received by the server, that’s the time for the server to make use of the
environ dictionary, the first argument the server passes into the
application callable, can be a bit of a black box, as it’s used to pass in multiple different pieces of data, which vary depending on the specific server and configuration. Included in this list are HTTP/request data, (method, query string, content length, content type, port number, headers, etc.), operating system environment variables (such as user defined private keys), WSGI variables (version number, input and error streams, process and thread data), and additional server variables.
In general, all the data that Django needs to receive from the server comes through the
environ dictionary. This, along with the
start_response callable, are the two required values that the server passes to Django. We’ll return to
start_response in a moment when it’s time to return the response to the server, but now Django has everything it needs to process the request, meaning we’ve completed step 2.
Turning a request into a response
While this may seem like the most important step, from WSGI’s perspective how this happens doesn’t matter. This is a standard part of developing any interface. By determining the rules of how two systems interact, but not how either one of them accomplishes their part of the contract, both systems have fewer dependencies and are free to make changes to their internal workings as long as they keep up their sides of the contract.
WSGIHandler.__call__() function listed above (aka the
application callable) also doesn’t care how the rest of Django handles the request. This is because its job is merely to facilitate Django’s side of WSGI, and it hands off request processing to other specialized functions. This is why it makes sense for the
__call__() function to have only the following lines of code for request handling:
request = self.request_class(environ) response = self.get_response(request)
In the first line, a designated function parses the
environ dictionary and separates out the HTTP request-specific data. The second line takes in the request object, and hands it off to
get_response to do the work, receiving a response object that’s ready to be returned to the server.
We now have our response. More specifically, the Django-defined
application callable function that the server has called has it’s response data. This isn’t actually an HTTP Response as far as the internet is concerned. It’s just a response object. The server handles the actual HTTP side of things, and then sends and receives data that’s formatted in a standard Python manner. Django doesn’t ever interact with the true HTTP request, just an abstraction of it.
We’ll cover the inner workings of
get_response in a future post, but for now step 3 is complete.
Returning the response to the server
Now that we have our response object, it’s time for Django to use the
start_response callable (passed into the
application callable by the server) to begin passing the response back.
This primarily consists of two steps:
start_responsewith its two required arguments,
status(e.g. “200 OK”) and
headers, which is list of header tuples.
- Returning an iterator containing the response body.
This might seem like a strange way of returning data to the server. Why doesn’t
start_response just take a third required argument called
body, thus simplifying the whole process?
For many cases, such as returning a simple web page, that would work. In fact, for simple responses, the iterator returned to the server is just a one-element list. But for larger amounts of data, or streams of data, this isn’t feasible. The server needs a way of getting the data in smaller blocks, processing the block in whatever manner is necessary, and then asking Django for the next block.
The iterator is what makes this possible. The most basic form of iterator is a list, which you’re probably familiar with:
for i in [1,2,3]: # Iterating over a list. print(i)
Under the hood, an iterator is just a class that has
__next__ methods. With a list the
__next__ method returns the next local value, which is already part of the list object’s state. But there’s no rule saying that the data returned by
__next__ has to already be a property of the object instance. The only requirement is that a block of data get returned.
This flexibility is what makes the iterator such a useful tool for Django to pass back to the server. If the response is simple, the iterator need only be a one-element list. But if there is more data, the iterator can be a more complex object that implements
__next__ by pulling data from the buffer that Django placed it in. From the server’s perspective, it makes no difference.
Now that the response has been returned to the server, Django doesn’t need to worry about it. The request has successfully been processed and a response has been returned. All Django has to do is wait for the next request to come in so the process can start all over again.