Django’s setup
method in the primary init.py file consists of a whopping nine lines of code; five if we don’t count the import lines.
def setup(set_prefix=True): from django.apps import apps from django.conf import settings from django.urls import set_script_prefix from django.utils.log import configure_logging configure_logging(settings.LOGGING_CONFIG, settings.LOGGING) if set_prefix: set_script_prefix( '/' if settings.FORCE_SCRIPT_NAME is None else settings.FORCE_SCRIPT_NAME ) apps.populate(settings.INSTALLED_APPS)
Hidden behind these few lines of code is a surprising amount of complexity, and by studying them we can gain a better understanding of how Django’s developers approach design and architecture, as well as a deeper understanding of Python’s built-in functionality.
Today’s post is the first of a four-part series on how Django’s setup process works.
1. How Django accesses project settings.
2. How Django and Python manage logging.
3. Why Django allows the script_prefix
to be overridden and how this is used.
4. How Django populates and stores applications.
You may notice that each post in this series corresponds to an import
line in the setup
function. This is because all four of these pieces rely on quite a bit of plumbing behind the scenes, and it’s exactly that plumbing that makes them so interesting.
When setup gets called
If you read the last post on how Django uses WSGI in its communication with the web server, you would have seen the following code from the wsgi.py file.
import django from django.core.handlers.wsgi import WSGIHandler def get_wsgi_application(): django.setup(set_prefix=False) return WSGIHandler()
We focused on the line return WSGIHandler()
and the ensuing functionality. This series is focused on the preceding line, django.setup(set_prefix=False)
meaning that everything that happens in this series takes place prior to the functionality described in the previous post.
Settings
Within the init.py
file, there are two lines of code that we care about for the current topic.
from django.conf import settings ... configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
The first line imports settings
from Django’s conf
package (i.e. config), and the second passes two of that object’s attributes to the configure_logging
function.
At first this looks simple; surely settings
is just some simple object pulled from a settings or config file, and now Django is looking up some pre-existing attributes.
Not quite.
By navigating to the /django/django/conf/__init__.py
file, we see that this file consists of three separate class definitions: LazySettings
, Settings
, and UserSettingsHolder
. The actual settings variable isn’t defined until the very last line of the file, where we find the following.
settings = LazySettings()
I encourage you to take a moment and look at the code briefly, as it’s surprisingly involved.
This a pattern we will see often in Django, in which an import object is actually just a variable set to the value of some object from the relevant package, and not necessarily the name of an actual class.
Lazy Settings
The LazySettings
class has the following profile (I’ve removed comments and replaced some areas we won’t be focusing on with ‘...
‘).
ENVIRONMENT_VARIABLE = "DJANGO_SETTINGS_MODULE" class LazySettings(LazyObject): def _setup(self, name=None): settings_module = os.environ.get(ENVIRONMENT_VARIABLE) ... self._wrapped = Settings(settings_module) def __repr__(self): ... def __getattr__(self, name): if self._wrapped is empty: self._setup(name) val = getattr(self._wrapped, name) self.__dict__[name] = val return val def __setattr__(self, name, value): ... def __delattr__(self, name): ... def configure(self, default_settings=global_settings, **options): if self._wrapped is not empty: raise RuntimeError('Settings already configured.') holder = UserSettingsHolder(default_settings) for name, value in options.items(): setattr(holder, name, value) self._wrapped = holder @property def configured(self): return self._wrapped is not empty
High level observations:
1. This class inherits from LazyObject
.
2. There’s no __init__
function, as it’s getting inherited from LazyObject
.
3. Of the seven methods, four are magic methods, one is considered private (via the preceding underscore in _setup
), one has the @property
decorator, and only one is a non-magic, public method.
What we don’t see are any attributes getting set. That makes sense if it’s inheriting the __init__
function from LazyObject
, so let’s take a look there and see if it’s setting the settings.LOGGING_CONFIG
and settings.LOGGING
attributes that get addressed in init.py
.
_wrapped = None def __init__(self): self._wrapped = empty
Nothing, except for the _wrapped
private attribute.
What’s going on?
In short, LazySettings
isn’t actually the main settings object. It’s just a wrapper around another object that holds the actual settings. If that’s confusing, we’ll first look at how this interaction works, and then discuss why it’s designed this way.
Get Attribute
The key method for understanding how setup works within Django is LazySettings.__getattr__
.
def __getattr__(self, name): if self._wrapped is empty: self._setup(name) val = getattr(self._wrapped, name) self.__dict__[name] = val return val
This method is called when an attribute that has not yet been defined gets accessed. This is exactly what happens with the settings.LOGGING_CONFIG
attribute. When accessed within init.py
, it hasn’t yet been set Accessing it results in a call to LazySettings.__getattr__
, which checks if LazySettings._wrapped
is empty. If so, it calls LazySettings._setup
, causing all settings to get initialized.
Once LazySettings._wrapped
has been defined, the value is looked up from the LazySettings._wrapped
object, assigned to the internal LazySettings.__dict__
attribute, and returned to the client. The __dict__
attribute is the dictionary that holds object-level attributes, which makes the following lines of code functionally equivalent.
self.x = 5 self.__dict__[x] = 5
Django is looking up an attribute from whatever object is stored at self._wrapped
, then saving that value as an attribute of the LazySettings
object. Future lookups will then be able to access the LazySettings
attribute directly, and won’t go through the LazySettings.__getattr__
method.
LazySettings._setup
The LazySettings._setup
method does the following: It looks up the DJANGO_SETTINGS_MODULE
environment variable, then passes this path into the __init__
method of the new Settings
object. This object is then assigned to the LazySettings._wrapped
attribute referenced earlier.
def _setup(self, name=None): settings_module = os.environ.get(ENVIRONMENT_VARIABLE) ... self._wrapped = Settings(settings_module)
This whole flow is a bit confusing, so let’s take a look at it visually.
- After the initial import, we have an empty
LazySettings
object.
- During the first lookup, after
self._setup(name)
has been run, but beforeself.__dict__[name] = val
is executed.
- After
self.__dict__[name] = val
has been executed.
Non-lazy settings
We now have a LazySettings
object that acts as a wrapper around a Settings
object, which isn’t instantiated until one of the settings is actually accessed. Let’s take a look at the inner Settings
object, to complete the picture of how settings get stored.
As shown in the (abridged) snippet of code below, the actual Settings
object is quite simple. There are two primary parts of the __init__
method.
- Iteration over the
global_settings
, defined in the/django/django/conf/__init__.py
file. This is the default Django settings module. Individual settings are saved as attributes of theSettings
object. - Import the user-defined
settings_module
path that was received as an argument, and iterate over these values. These may override settings previously set via theglobal_settings
module, and each one is added to theSettings._explicit_settings
set, to keep a record of all user-defined settings.from django.conf import global_settings ... class Settings: def __init__(self, settings_module): for setting in dir(global_settings): if setting.isupper(): setattr(self, setting, getattr(global_settings, setting)) # store the settings module in case someone later cares self.SETTINGS_MODULE = settings_module mod = importlib.import_module(self.SETTINGS_MODULE) ... self._explicit_settings = set() for setting in dir(mod): if setting.isupper(): setting_value = getattr(mod, setting) ... setattr(self, setting, setting_value) self._explicit_settings.add(setting) ... def is_overridden(self, setting): return setting in self._explicit_settings ...
Accessible Settings
Now that Settings.__init__
has run, the individual settings are accessible within the rest of the project and framework. In fact, if we return to the initial two lines of code where we first encountered settings, this entire process takes place while settings.LOGGING_CONFIG
is first being looked up. When settings.LOGGING
is accessed, the LazySettings
object simply grabs the attribute from an already-populated Settings
object, stores it as an attribute of its own, and returns the value to the user.
If LOGGING
is accessed a second time, the __getattr__
method will be skipped entirely, and the attribute will be pulled from the LazySettings
object just like any other attribute.
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
Wait, but why?
We’ve covered how the settings are accessed, but we haven’t taken a look at why they work this way. It isn’t the most simple method for accessing settings. Why not just create a Settings
object without a wrapper, which can be accessed directly?
Additionally, why was the UserSettingsHolder
class defined in this file if it wasn’t used for anything thus far? The answers are related.
Django supports the ability of different modules to configure settings manually when they are run separately from the rest of the Django app. The documentation gives the example of using Django’s template system by itself. In this case, the module could include from django.conf import settings
and access settings.{attribute}
directly, thereby initiating the same configuration process described above in the __init__.py
file. The problem is that the separate module might not want this group of settings, and it may not want to rely on DJANGO_SETTINGS_MODULE
environment variable.
Enter LazySettings.configure
. This method allows custom, manual configuration of the settings. As shown below, the configure method receives the settings as parameters and passes them to the UserSettingsHolder
object. In this case the self._wrapped
attribute is stored as a UserSettingsHolder
object rather than an instance of Settings
, simply because the methods of retrieving and accessing settings are different in the two cases, and are best served by distinct classes.
def configure(self, default_settings=global_settings, **options): if self._wrapped is not empty: raise RuntimeError('Settings already configured.') holder = UserSettingsHolder(default_settings) for name, value in options.items(): setattr(holder, name, value) self._wrapped = holder
This capability is why the LazySettings
class is necessary (or at least a useful solution). Django only allows the settings to be configured one time. If the settings were configured by default every time, then specific modules would be unable to custom-configure their settings. The existing solution may seem overly-complicated, but it allows a somewhat unique set of rules to be enforced with minimal overhead for the user.
- Settings are guaranteed to only be configured one time.
- Settings can be set manually if the developer would like.
- Otherwise, the default configuration process takes place, and the developer doesn’t need to manually configure anything.
Conclusion
Settings may seem like an unlikely candidate for a (relatively) deep dive, but taking a closer look gives us a more nuanced understanding of how Django actually works, how it utilizes Python’s built in functionality, and perhaps more importantly, why Django is built in this manner. In building a fully featured framework, Django’s core developers clearly took great time and care to design solutions that make usage as simple as possible for developers and users. But simple and flexible end-products sometimes necessitate fairly complex architectures, and we see that here.
Thanks for reading, and tune in next time when we dive into the (actually) exciting world of logging, as we continue this tour through Django’s setup process.
Takeaway Questions: Does this solution seem like it adequately fits the problem, or does it seem like over-engineering? Assuming the purpose was for the end product to be simple and flexible, does it succeed in this job?
Feel free to leave your thoughts in the comments below.
Very interesting blog. I am looking forward to read the whole series.
nice!:)