If you are new to software development, then there is a long road before you. But Django is a good choice, since it is a well established and very good documented framework.
First learn Python, then some HTML and CSS. Then Django and SQL.
After you learned the basics (Python, some HTML, some CSS), then use the Django tutorial.
Avoid ReactJS or Vue, since you might not need them. First start with the traditional approach: Create HTML on the server and send it to the web browser.
If you want to use a CSS library, then I recommend Bootstrap5.
You can start with SQLite, but sooner or later you should switch to PostgreSQL.
My hint: Don't dive too deep into JavaScript. It is less important than most people think.
According to Vitor Freitas: "always replace the default User model" What You Should Know About The Django User Model
It is important to understand the difference between a project and an application in the context of Django. Please read Projects and Applications
I always try to keep the project small. The project is a small container. One projects contains several apps. The project contains settings, but almost no code.
I always call the project mysite, like in the Django tutorial.
This has the benefit that the env var DJANGO_SETTINGS_MODULE is always "mysite.settings" in all my personal projects.
Errors get silently ignored by the django template language. I avoid it.
I like to create small html snippets with format_html().
I love Python methods. They are simple to write, easy to understand and exceptions show a nice stacktrace.
But if you use the template language, you can not easily mix custom template tags with custom template filters. Example: https://stackoverflow.com/a/12249207/633961
Next thing why I don't prefer to create HTML with Python:
def my_special_method(self, obj):
return ...
my_special_method.short_descript = 'super name'
Above is a common pattern. Now try to access 'short_description' via the template language:
{{ obj.my_special_method.short_description }}
This does not work, since the template language does this obj.my_special_method().short_description. Grrr
That's why I prefer format_html() in Python code.
Don't use (or try to understand) the Django cycle templatetag. Today you don't need to alter the css class to create cebra-tables. Vanilla CSS is enough.
If you are new to a big project, you might not easily see which view function does handle an URL.
You can use resolve() to get the answer:
import django
django.setup()
from django.urls import resolve
print(resolve('/foo/bar/'))
--> ResolverMatch(func=foocms.views.bar_handler, args=('/foo/bar/',), kwargs={}, url_name=foocms, ...)
Now you know that foocms.views.bar_handler will handle requests to the URL /foo/bar/.
Sometimes you want to check some code against the production DB just for testing.
Getting changes through CI would take too long.
You can connect you local Django to the production DB, and make the connection read-only.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': ...
'USER': ...
'PASSWORD': ...
'OPTIONS': {
'options': '-c default_transaction_read_only=on'
}
}
}
Now you can run your local code connected to the production DB, and be sure that you don't break things.
See: https://stackoverflow.com/a/66986980/633961
foo/start-overview.html
<table>
<tr><th>Col1</th>...</tr>
...
foo/end-overview.html
</table>
===> NO!
Keep the opening and closing tag together!
foo/overview.html
<table>
{% include 'foo/overview-heading.html' %}
...
</table>
settings.py
from dotenv import load_dotenv
from distutils.util import strtobool
load_dotenv()
DEBUG = strtobool(os.getenv('DEBUG'))
...
The Django Debug Toolbar is really useful.
For small projects it is even fine to enable it in production via SHOW_TOOLBAR_CALLBACK. Example: The toolbar gets shown, only
if the URL contains _debug=SomeMagicString.
Third party panel for DDT:
I use the library pytest-django.
If you want to get an exception (as opposed to an empty string) if a template variable is unknown, then you can use this config:
pytest.ini:
[pytest]
DJANGO_SETTINGS_MODULE = mysite.settings
FAIL_INVALID_TEMPLATE_VARS = True
FAIL_INVALID_TEMPLATE_VARS causes the rendering of a template to fail, if a template variable does not exist. I like this. See Zen-of-Python "Errors should never pass silently."
See pytest-django docs "fail for invalid variables in templates
django_assert_num_queries is handy, if you want to ensure that the number of SQL queries does not increase over time.
Selenium automates browsers. It can automated modern browsers and IE. It is flaky. It will randomly fail, and you will waste a lot of time. Avoid to support IE, and prefer to focus on development.
Google Trend "Selenium" is going down.
I heared that PlayWright is modern solution to automate Chromium, Firefox and WebKit with a single API.
If possible, I test methods without a http request in a small unittest.
On the other hand I would like to know if the html forms are usable by a web-browser.
I like to test my django forms like this:
- I use
reverse()to get an URL - I use the pytest client to get a http response
- I use html_form_to_dict() to get a dictionary of the form which is in
response.content - I set some values on
data - I use the
client.post(url, data)to submit the form
For an example, please see the README of html_form_to_dict()
Software test fixtures initialize test functions. They provide a fixed baseline so that tests execute reliably and produce consistent, repeatable, results.
I don't need Django Fixtures. If I need data in a production systems, I can use a script or database migration.
If I need data for a test, then I use pytest fixtures
It is unfortunate that the term "fixture" has two completely different meanings in the django terminology. I never use django-fixtures, but use pytest-fixtures daily.
And I don't see a reason to use a library like factory-boy. I prefer pytest fixtures. Pytest fixtures and Django-ORM gives me all I need.
I avoid creating random data. I don't use libraries like faker. I want things to be repeatable and predictable.
I know coding and testing your changes manually via runserver and ctrl-r in browser is like a magnet. Even if you know that test-driven-development is better you will get trapped in the "edit, ctrl-r" loop.
What do you mean with "edit ctrl-r" loop? It works like this: You edit your source code, and then your use the browser to see if your changes have the desired effect. And looking at your changes happens by using your browser.
This works, but has one draw-back: After you got your code change done, you have no automated test. You did manually testing, and it is likely that someone will break the things you implemented in your change.
This loop makes you faster in the long run: "Edit, run automated test".
The file models.py is a mixture. A mixture of schema definition and logic. That's handy and makes you fast at the beginning.
But don't put too much logic into this file. Your subclasses of Model should only contain basic methods.
Business Logic, creating HTML and other stuff should live somewhere else.
I usualy create a file per model: If the model is called Foo, then I create a file called fooutils.py which contains
methods which get instance of the class Foo as first argument.
Example for a model called Customer:
# customerutils.py
def create_weekly_report(customer):
...
Django Typed Models brings Single Table Inheritance to Django.
I like it for use-cases like this: Imagine you want to track the changes which get done by a user. The user creates an account, then the user adds his address. Later he creates an order, ....
You could create a model like this:
class Log(models.Model):
user = models.ForeignKey(User)
time = models.DateTimeField(auto_add_now=True)
action = ....
data = models.JSONField() action could be a CharField with choices, or a ForeignKey to a Model which contains all the possible choices as rows.
data stores the changes which fit to the specific action.
Example: the action "Order Created" needs a link to the relevant offer.
Every action has its own data schema.... Things get fuzzy if you use JSON.
OR you could use Single Table Inheritance:
class Log(TypedModel):
user = models.ForeignKey(User)
time = models.DateTimeField(auto_add_now=True)
class OrderCreatedLog(Log):
offer = models.ForeignKey(Offer)This way you don't need JSON, you can use database column to store the values.
I wrote a small Check HTML Middleware, so that typos in HTML get detected soon:
django-fieldsignals is a handy library, so that you can easily receive a signal if a field of a model has changed.
If you follow the current hype, you get the impression that web applications must be build like this:
There is a backend (for example Django) which provides an http-API. This API gets used by a JavaScript application written in Angular, React, or Vue.
Wait, slow down.
I think there is a simpler and more efficient way to develop an web application: Just create HTML on the server side with Django.
To make your application load/submit HTML snippets (to avoid the full screen refresh) you can use htmx.
This way you have a simple stack which gives you a solid foundation for your application.
from django.utils.timezone import get_current_timezone
get_current_timezone().localize(datetime.datetime(1999, 1, 1, 0, 0, 0))
--> datetime.datetime(1999, 1, 1, 0, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CET+1:00:00 STD>)
From: Create timezone aware datetime objects
You want to create one HTML page which contains three forms. The Django forms library is great, but it does not solve this problem for you.
You could use Prefixes for forms and put your three django-forms onto one big page. Depending on the context, this often feels too heavy.
That's where htmx can help you: With htmx you can load/submit html fragments easily. No need to write a SPA (Single Page Application), but if you want to, htmx gives you all you need.
To make your HTML look good on mobile and desktop I use Bootstrap5.
The Python ecosystem is much bigger than the Django ecosystem.
For example you want to visualize data. Don't search for a django solution, search for a Python solution. For example: Holoviews
Or use a JS based solution. For example d3
Imagine you write a page so that the user is able to edit his/her address. You use request.user and everything works fine.
Now you want to make the same form available to a superuser, so that the superuser can edit the address of a user.
Now things get mixed up. Because request.user is not longer the user of the address ....
You can avoid the confusion if you avoid request.user and instead require that the caller explicitly gives you
an user object.
It is perfectly fine to store the current user in a threadlocal variable. But just use this global user for permission checking.
Up to now I use a global request middleware. But I am not happy with it. I will use a global user middleware in the future.
This way I can check for permission without passing the request to every method which will be called during handling the http-request.
But making the request available via a global variable is too much implicit input to methods.
Don't give a customer is_superuser. Give him is_staff and Permissions.
Sooner or later you want to add functionality which should be only available to you (the SaaS maintainer).
In the past I had the whole stack installed in my local development environment (Apache or Nginx/Gunicorn), but
I don't do this any more. The runserver of Django is enough for development. You usualy don't need https during development.
This contradicts the guideline that the development environment and the production environment should be as similar as possible.
The runserver reloads code fast, which is great for a fluent "edit test" cycle.
I develop on Ubuntu Linux with PyCharm.
But I use PostgreSQL on production and for development. If you use the same username for your PostgreSQL-user like for your Linux-user, then you don't need to configure a password for the database.
I use a cheap VPS from Hetzner. Every system I run on the VPS has its own Linux user. Every Linux-User has a virtualenv in $HOME, which gets activated in .bashrc. This is handy if you want to check something with SSH.
I run gunicorn webserver via an Systems like explained in the Gunicorn Deploying Docs
This way I can run several systems on one VPS. This means there are N gunicorn processes.
As reverse proxy and https endpoint I use Nginx.
Sooner or alter I will switch to containers, but at the moment my current setup works fine.
You should understand that's the job of the webserver to provide https (Certificates ...) or to compress the responses.
It makes no sense to use the Django GZipMiddleware.
The setting SECURE_SSL_REDIRECT: I think redirecting from http to https should be done by the web-server, not by Django.
PostgreSQL can do full text search, and Django supports it: PG full text search
You make your life easier, if you avoid a second system (for example ElasticSearch).
First I run pg_dump, then timegaps to remove outdated dumps, then rsync to a second VPS.
Use django-allauth
Use the library WhiteNoise, even during development
Asking questions is important. It pushes you and others forward.
But you need to dinstinguish between vague and specific questions.
Ask in the Django Forum or in a Django Facebook Group. For example Django Python Web Framework
For example: "Which library should I use for ...".
Ask on Stackoverflow
The best questions provide some example code which can be executed easily. This increases the likelihood that someone will provide an answer soon.
If your code creates a stacktrace, then please add the whole stacktrace to the question.
Unfortunately not that cool like state of js or state of css, but the Django Survey gives you a nice overview, too.
I like the Django Forms Library. I don' use third party packages like crispy forms.
Rule of thumb: Don't access request.GET or request.POST. Always use a form to retrieve the values from the request.
I know Adding extra manager methods. But I don't like it.
I prefer to write a @classmethod if I want to return a custom querysets.
Same for subclassing QuerySet. I don't think this is useful. A classmethod is easier to understand (my POV).
Why do I prefer classmehods? This is a gut feeling. Maybe because Python is bigger than Django. There are more people who know Python, than people who know Django. If someone knows Python, but not Django, then he would understand a classmethod, but not a custom django manager class.
My POV: Use real ForeignKeys, not GenericForeignKeys
"Function based Views" vs "Class based Views"?
Since I switched to htmx.org I prefer FBV.
You don’t need to learn any of the CBV APIs - TemplateView, ListView, DetailView, FormView, MultipleObjectMixin etc. etc. and all their inheritance trees or method flowcharts. They will only make your life harder.You don’t need to learn any of the CBV APIs - TemplateView, ListView, DetailView, FormView, MultipleObjectMixin etc. etc. and all their inheritance trees or method flowcharts. They will only make your life harder.
Source: Django Views - The Right Way by Luke Plant.
- get_object_or_404() is handy.
If you use the default of SESSION_COOKIE_SAMESITE, then you don't need a CSRF token.
sorl-thumbnail is a handy library to create thumbnails.
During development on your local machine you get a nice debug-view if there is an uncaught exception.
On the production system you get a white page "Server Error (500)".
And of course, you want to know if users of your site get server error.
Personally I prefer simple open source solutions to free and great commercial solutions. But in this case I settled with Sentry.
It is simple to set up, is free and shows all uncaught exceptions in an easy to use web GUI.
And Sentry is a Gold Corporate Member of the Django Software Foundation.
Sentry won't help you, if there is something broken and your http server does not start. To be sure that your site is running you can use a free service. There are several, I use https://uptimerobot.com/
They check your site every N minutes from outside via https.
New content, new URL.
django-storages AWS_S3_FILE_OVERWRITE = False forces you to use create+delete instead of updating content.
You can use Lighthouse (via Chrome) or PageSpeed Insights (Google) to check your page.
- OpenSlides OpenSlides is the all-in-one solution for running your plenary meetings and conferences.
- Taiga Agile project management platform. Built on top of Django and AngularJS
- Saleor e-commerce plattform
"There are only two hard things in Computer Science: cache invalidation and naming things."
In ModelForm it is called "instance". In a class-based-view it is called "object". In Admin templates it is called "original". It would be nice to have one term for the thing.
Don't ask me why the reverse() method is called "url" in Django templates.
The Django template language hides errors. I prefer format_html()
Adam has a great article about How to Make Django Raise an Error for Missing Template Variables.
The term "Field" has two meanings, and this creates confusion. There are ModelFields and FormFields.
Template language: allow for {% if foo in ["one", "two", "three"] %}. Related: check for presence in a list django template
Parentheses in expression in the template language would be nice. Related docs Boolean operators
Don't change old migrations which you already pushed to a central repository. It is likely that someone already pulled your changes into his development environment. This developer will have trouble if you change the old migration. Create a new migration which fixes the error of the old migration.
If you develop in a team you will sooner or later get trouble with your migrations, because two developers create a new migration in their branch. The CI for each branch is fine, but after merging both to the main branch you have two migrations with the same number. Grrr
django-linear-migrations helps you. At least you know during merging that there is a conflict.
The solution is simple:
It does this by creating new files in your apps’ migration directories, called max_migration.txt. These files contain the latest migration name for that app, and are automatically updated by makemigrations.
Big thank you to Adam Chainz for this package.
Since the ORM solves most use-cases many developers don't use raw SQL queries in Django. This does not mean SQL is "evil". SQL is great!
With the help of Subquery and OuterRef complex queries are possible. But
nevertheless, I think SQL is more readable than ORM queries containing OuterRef(OuterRef(...)).
If a raw SQL query is easier to understand than the ORM counterpart (and there is a test for it), then go for SQL.
But of course you should be aware of SQL-injection and use parameters like documented:
Person.objects.raw('SELECT * FROM myapp_person WHERE last_name = %s', [lname])
Espescialy in data migrations I often prefer SQL to ORM.
As a Django developer you might feel more comfortable with the ORM than with SQL. I know. But look at the whole plant earth, then there are far more people who know SQL than people who now the Django ORM.
Imagine you have private files which you don't want to serve via a CDN. You could use a hard to guess random string in the URL, but that's not a real solution.
You can use the "x-sendfile" approach. This way you can do the authentication and permission checking in your Python code, and then let the webserver (for example Nginx) handle the slow work of sending the file over the wire.
Setting the appropriate http headers is not hard. But you can use django-sendfile2, too.
The inner feedback loop:
- Edit
- Save
- Compile
- Run
- Check result
With Python and a modern IDE "Save" and "Compile" are not needed.
Nevertheless it takes some time to see the result.
If you are fiddling with HTML+CSS you might be faster if you edit the HTML+CSS directly in the browser. For example in devtools (Chrome).
In most cases step "5. Check result" means to see if a test was successful or not. Test-Driven Development makes you faster in the long run.
User.objects.filter(groups__isnull=False)
Related: https://stackoverflow.com/questions/54367178/django-orm-all-users-which-have-at-least-one-group
I am unsure wheter annotate and aggregate are cool features or not. Every time I use
them, I know the SQL which I would write. Then I need to translate the SQL in my head
to the Django syntax. Is this useful? I would be faster if I could write SQL directely.
Of course I could, but I don't, since writing raw SQL is frowned upon in the Django context....
User.objects.filter(...).query
python-diskcache is an replacement for the FileBasedCache. For small projects you might not need a Redis server.
Django has an excelent hook-system, but many don't know it: Signals.
You can use the return values of signals handlers.
This is handy if you have a core-app with several plugins.
- Create a custom signal in the core-app.
- Implement a signal handler in the optional plugin-app.
- Call the signal handler from the core-app.
- Use the return values of `Signal.send()
Docs:
Both send() and send_robust() return a list of tuple pairs [(receiver, response), ... ], representing the list of called receiver functions and their response values.
django.contrib.postgres has some nice features.
For example:
- CICharField Case-insensitive CharField
django-pghistory record changes to models via PostgreSQL triggers.
I like to create HTML with format_html()
I like to have all valuable data in PostgreSQL. That's why I like to tasks in the DB (and not in Redis or RabbitMQ).
You want to list some hyperlinks separated by comma?
{% for obj in mylist %}
<a href="{{obj.get_absolute_url}}">{{obj}}</a>{% if not forloop.last %}, {% endif %}
{% endfor %}
See "for" in Built-in Template Language
You can easily create data for unittest with pytest-fixtures.
But these DB rows are temporary and only for unit-testing.
You want a demo system populated with some dummy data if you want to use your django system with the web-browser.
For small projects, this is a minor issue. You can create some dummy data by hand.
Or you can dump the production database and restore it in your development DB. But wait! This is unprofessional. This might work for small projects, but for bigger projects this is against data protection regulations. Developer should not see the data of the customers.
Of course you can help yourself with a script.
But it would be really cool to have a plugin-system so that every app can contribute to setting up a demo data.
Investing time into this makes sense.
Related: https://code.djangoproject.com/ticket/33467
I am not looking for libraries like factory-boy, faker or django-seed. I think every app should create the data the way the app wants to create the data.
The interesting part is the dependency management. If app "cms-plugin" needs the data of "cms-core", then you need a way specify the dependency. Where "dependency" means, that the cms-plugin uses the primary-key of a row created by cms-core as foreign-key.
For example every page needs a category. Imagine there are two plugins. cms-plugin1 creates a category, and cms-plugin2 depends on the existance of this category.
Every app should provide N demo data creators. For example the cms-core could provide demo-data-users, demo-data-countries. An other app should be able to depend on demo-data-users without depending on demo-data-countries.
Row based (AKA "per object permissions") are not easy. There are several solutions:
django-gardian Drawback: Slow. See Performance
Use get_queryset() in the admin. This is fast and flexible. But does not distinguish between "read permissions" and "write permissions".
Video of Madelaine Boyd about row based permissions
The django admin has filters so you can filter your models.
I would like to have set operations:
Create a set with the admin filters. Store this list.
Then do a send filter. Then do set operations on both lists:
- A minus B
- A plus B
- B minus A
- Intersection of A and B.
- ....
Imagine you want to render a hyperlink, if a user is allowed to access this view. But if the user is not allowed to access the hyperlink, then you want to display some text.
Of course you can handle this on your own. This is easy if the view and the code which creates the hyperlink are under your control. Django gives you all the tools to solve this.
But there is no generic way of doing this. If the view is third party code, then it is likely that permission checking gets done inside the view. Then you can't check if the user is allowed or not access the page without calling the view.
If you have a request, then it is easy: request.build_absolute_uri().
But in a command line script, you don't have a request.
AFAIK there is no standard way. See How can I get the URL (with protocol and domain) in Django (without request)?
In most cases, this object will not have been saved to the database yet, so if it uses the default AutoField, it might not yet have a value for its primary key field.
Django FileField.upload_to docs
It would be great if I could use the PK for storing the file.
You can use string_if_invalid, but this means you can use a "default" for the string. See: Django string_if_invalid and default
For pytest you can use FAIL_INVALID_TEMPLATE_VARS = True but again, "defaults" does not work.
If the project uses htmx, this is not an issue. Because with htmx I often write small methods returning small snippets. This means I use format_html() instead of the template language.
You can't call a method with arguments via the template language. I wish you could. Related How to call function that takes an argument in a Django template?
In most cases you don't your files to override each other. This means instance-1 should be allowed to have a file called image.png and instance-2 should have a file with the same name (but different content).
foo_file = FileField(_('Foo File'),
upload_to=lambda instance, filename: datetime.datetime.now().strftime(
# use random to avoid clashing filenames
'%Y/%m/%d-{}/{}'.format(random.randint(100000, 999999),
filename)),
unique=True,
max_length=1024, blank=True, null=True)
Unfortunately it is not possible up to now to use instance.pk in "upload_to".
Don't sell the Admin-Interface to your customer.
This might make you faster on day-1, but in the long run it will slow you down.
django-perf-rec is like Django’s assertNumQueries on steroids. It lets you track the individual queries and cache operations that occur in your code. It stores a YAML file alongside the test file that tracks the queries and operations.
Celery is very common task-queue.
Nevertheless it is big and a second data-store. I prefer to have all data in one DB.
With PostgreSQL you can write simple task-queues yourself. Just create a table and and add new rows for each task.
Then you can use SELECT ... FOR UPDATE SKIP LOCKED in your worker processes.
This should work for most cases. This means there is no need for a second data-store and the huge celery library.
You can use LISTEN, to process new data immediately.
Dan Palmer - Scaling Django to 500 apps
In my experience django.contrib.sites creates more confusion, then it brings value.
As soon as you use one database for several sites, you need to make sure for every request that you apply the matching filters, so that only the corresponding database rows get used.
Which site to choose if there not http request, like in CronJobs?
This makes software development complicated every day.
Running two systems makes more sense for me. Running two system is more work at day-one, but once it is set up and CI/CD is automated, you are done.
I have seen (non-django) projects which use a single database for all of their customers, although each customer must only see his data. There was no data shared between the customers, so that there was no reason for this architecture. The single-DB architecture slowed down the development daily at several layers (development, deployment, fear of bringing the single DB down, ...)
Don't use select_related() except you check the performance with a benchmark and write a unit-tests which
checks that the number of SQL queries does not increase.
I have seen code where developers seem to have used select_related() randomly.
Maybe an annotated query does provide even better performance.
Nobody knows, except you benchmark it, and leave a link to the benchmark in the source code.
If you get the above error message, it is sometimes hard to understand what is missing.
Step 1: Create a test to reproduce the error.
Step 2: Alter the django source code, to give you a better message:
File django/forms/formsets.py
if not form.is_valid():
raise ValidationError(
- _('ManagementForm data is missing or has been tampered with'),
+ _('ManagementForm data is missing or has been tampered with xxxxxx prefix={} errors: {}'.format(
+ form.prefix, form.errors)),
code='missing_management_form',
)
Now you see what is missing.
Jeremy Bowman - Herding Ponies: Coordinating and Automating Django Upgrades Across 100+ Repositories
Not all invalid states can be avoided by database contstraints. Nevertheless you want to check that everything is in a valid state.
You could write a custom manage command for this. See custom management commands.
But I think it makes more sense to write a view which returns a HTML page and a corresponding http status code.
HTML gives you much more freedom. For example you can create tables, which you can't if you use text output.
If you use get_or_create() (or update_or_create()), then you most likely want your
model to have a corresponding unique constraint.
If you don't have a unique constraint, and two http-requests get handled by your application, which
execute the same get_or_create(), then both http-requests will create a new row.
Now your state in the DB is broken, because the next call to get_or_create() with the same values,
will fail:
MultipleObjectsReturned: get() returned more than one FooModel -- it returned 2!
Examples:
You use FooModel.objects.get_or_create(foo=...), then the attribute foo on you
model needs to be unique.
If you use OtherModel.objects.get_or_create(my_col=..., my_other_col=...), then you need unique_together.
In the development environment open template/base.py and add a print statement:
class Template(object):
def __init__(self, template_string, origin=None,
name='<Unknown Template>'):
print('.............loading', name)