Django community: Community blog posts RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Django under the hood: validation - Loïc Bistuer
(One of my summaries of a talk at the 2016 django under the hood conference). Loïc has mostly worked on forms and the ORM. The main concerns with validation are: Enforcement. User experience Performance Convenience Some items go well together. Enforcement and user experience like each other. You don't want wrong data. And you want good feedback. Validation helps with that. But "user experience" and "performance" are harder to combine. Checks do cost time. Similarly "user experience" and "developer convenience". Why do you have to check anything on the backend when you already checked it on the front end? Extra work. Where to validate data? You can do it in the front end: javascript, html5/browser or in native code like phone apps. The nice thing is that it is fast and provides direct feedback. The drawback is that you have to do the same thing on the backend again, as you cannot trust anything coming in from the front end. You can also use forms and the django rest framework serializer. Designed for the task, but it is easy to circumvent. Similarly django views. You could do validation directly on the model. Only problem is that it isn't run by … -
Django under the hood: keynote about mental health in tech - Jennifer Akullian
(One of my summaries of a talk at the 2016 django under the hood conference). The trust equation: Trust = (credibility + reliability + intimacy) / self-orientation. If you want to build trust with someone, keep this equation in mind. She asked for a show of hands: "how many people have glasses or contact lenses?" Afterwards she asked "how many people have been told to just try harder instead using glasses?" Laughter, no hands went up. "Well, I have a mental illness and have been told to try harder. Just focus on happy things, just try to be more happy, etc..." She has bipolar disorder. Found in about 2.6% in the population. It is genetic. It takes on average 7 years for someone to be accurately diagnosed, which is a strangely long time. She showed a diagram of three neurotransmitters (dopamine, norepinephrine, serotin) that don't chemically function well for her and listed some of the medications she takes. "Would it feel as uncomfortable if I would talk about medicines for a body illness? No. Does it feel uncomfortable to talk about medicines for a mental illness? Yes." The reasson: there is a stigma on it. Everybody knows the problem, but … -
Django under the hood: modern javascript - Idan Gazit
(One of my summaries of a talk at the 2016 django under the hood conference). There's some negative sentiment around Javascript. It might be hard. You might be scared of it. It is much less elegant than python. Etc. With "modern javascript" he means "civilized javascript". So how to work with javascript without missing python terribly. There are a lot of reasons why javascript might feel scary. Callback hell. Weird prototyping instead of regular classes. Less syntactic sugar. But.... javascript is the only runtime that is shipped with every browser! How did we get here? Originally, javascript saw limited use to put snow on your screen during christmas time and some form validation. Then came google with a super-fast javascript engine, V8. Node (=server side javascript) is basically the V8 engine with some libraries. So you have the browser world and the node world. Packaging for the browser is by hand or with bower and so. Packaging for node is done with "npm". For a long time, "ecmascript 5" was the main javascript. A bit like python 2. Everybody supported it. in 2015 there finally came a new, improved version: "ES6", "ecmascript 6". They've now decided to bring out a … -
Django under the hood: custom database backends - Michael Manfre
(One of my summaries of a talk at the 2016 django under the hood conference). Tip: watch django in depth by James Bennett. The database backend is right there at the bottom of the lowest level. What does the database backend do? It sits between the Django ORM and the actual database driver. There's a PEP249, the DB-API 2.0 specification for python code to talk to the actual database driver. Django abstracts away many of the differences between databases. But not all databases are created equal, so sometimes supporting what django expects is hard. Michael maintains the microsoft sql backend and showed some of the differences. If you need a custom database backend, you could subclass an existing django database backend. There's a read-only postgres db backend that has only a few lines of code. But if you create one from scratch, you need to implement about 8 classes. The DatabaseWrapper talks to the PEP249 python database library. Important: the "vendor" string to help django do specific things when it uses your database. There are other attributes that tell django how to map simple queries to actual SQL. iexact, less than, stuff like that. CursorWrapper. This one wraps the database … -
Django under the hood: django at instagram - Carl Meyer
(One of my summaries of a talk at the 2016 django under the hood conference). Instagram is huge. He mentioned a number of fun facts with lots of zeros in them. Oh, and cat photo's. They have tens of thousands django instances. Instagram started in 2010. According to one of the founders, django was super easy set-up. There is one obvious way of doing things. There's a test framework build-in. A few months later, october 2010, they had 1 million users. In June 2011, instagram had 5 million users. All in the database. With database routers they could partition it over multiple servers. The django ORM was managing those 5 million users in the database just fine. But slowly the number of likes was getting too much. It needed custom sharding. They used postgres schemas, which is more like a "logical shard". They could then map those logical shards at will unto actual physical servers. In the end, they started a custom ORM to better handle the huge amount of sharding that they needed. The likes were moved over first, two years later the user data moved. The Django ORM is still used in places, but the huge data is … -
How to start developing a Django project?
Django is the most popular Python web framework to date. The tagline “For perfectionist with deadlines” perfectly sums up what you can expect if you start using it. You have control over every aspect of your application, but it require some serious commitment to wield that power. When in doubt check out the excellent documentation, which will definitely solve most of your challenges. This blog post is about how to start developing a Django Project on your computer. I am assuming you are using Linux operating system, specifically Ubuntu. Also, I had absolute beginners in mind. This is a technical tutorial about what exact commands you need to put in, to get up and running. That said, let's start: Basic Directory Structure First, create a directory you want your project in: mkdir -p DjangoTutorial/{media,static,virtualenv,database} (don't leave spaces in the last part) This command creates the whole directory tree for you: DjangoTutorial/ database/ media/ static/ virtualenv/ Navigate into DjangoTutorial, then initiate virtualenv with the following command: cd DjangoTutorial virtualenv –python=python3 ./virtualenv Virtual Environment will make sure, that you will have consistent environments when you deploy your application to different servers. You will install plugins and packages, and you will replicate those … -
Optimization of QuerySet.get() with or without select_related
If you know you're going to look up a related Django ORM object from another one, Django automatically takes care of that for you. To illustrate, imaging a mapping that looks like this: class Artist(models.Models): name = models.CharField(max_length=200) ... class Song(models.Models): artist = models.ForeignKey(Artist) ... And with that in mind, suppose you do this: >>> Song.objects.get(id=1234567).artist.name 'Frank Zappa' Internally, what Django does is that it looks the Song object first, then it does a look up automatically on the Artist. In PostgreSQL it looks something like this: SELECT "main_song"."id", "main_song"."artist_id", ... FROM "main_song" WHERE "main_song"."id" = 1234567 SELECT "main_artist"."id", "main_artist"."name", ... FROM "main_artist" WHERE "main_artist"."id" = 111 Pretty clear. Right. Now if you know you're going to need to look up that related field you can ask Django to make a join before the lookup even happens. It looks like this: >>> Song.objects.select_related('artist').get(id=1234567).artist.name 'Frank Zappa' And the SQL needed looks like this: SELECT "main_song"."id", ... , "main_artist"."name", ... FROM "main_song" INNER JOIN "main_artist" ON ("main_song"."artist_id" = "main_artist"."id") WHERE "main_song"."id" = 1234567 The question is; which is fastest? Well, there's only one way to find out and that is to measure with some relatistic data. Here's the benchmarking code: def f1(id): … -
Managing multiple Python projects: Virtual environments
Even Python learning materials that get into very advanced language features rarely mention some practical things that would be very helpful to know as soon as you start working on more serious projects, like: How to install packages written by others so that your code can use them, without just copying the files into your own project. How to work on multiple projects on one computer that might depend on different packages, and even different versions of the same packages, without them interfering with each other. The key concept that helps to manage all this is the "virtual environment". A virtual environment is a way of giving each of your Python projects a separate and isolated world to run in, with its own version of Python and installed libraries. It’s almost like installing a completely separate copy of Python for each project to use, but it’s much lighter weight than that. When you create a virtual environment named "foo", somewhere on your computer, a new directory named "foo" is created. There's a "bin" directory inside it, which contains a "python" executable. When you run that python executable, it will only have access to the python built-in libraries and any libraries … -
Django under the hood: Channels - Andrew Godwin
(One of my summaries of a talk at the 2016 django under the hood conference). Django channels was started by Andrew Godwin, best known for his work on South and django migrations. Channels might seem like magic, but it is not. He'll start by describing the problem and then the actual django channels work. The problem: the web is changing A lot of the web is becoming async. Web sockets and so. But it is not only websockets: long-polling, webrtc, MQTT, server-sent events. This doesn't match django's regular webpage behaviour very well. Python is synchronous. Only the latest python 3 releases have async build-in. But even then, Django is still synchronous at the core. Synchronous code is easier to write. Async is much harder. Only do it if you really have to. Synchronous code is much easier to reason about. Single-process async is not good enough, so you'll have multiple processes, threads and perhaps even machines. You'll need a proven design pattern that isn't too hard to reason about. This is no place to do something weird and new. Multiple people must be able to maintain it. And if you use it, you'll need many people who are able to … -
DSA ssh keys also deprecated in OSX Sierra
I've been using an ssh key for a long time. Back in the days when RSA keys were mostly commercial/proprietary and when DSA keys where the recommended method. When ubuntu 16.04 came out, I suddenly couldn't use my key anymore as ubuntu (or perhaps rather its updated ssh server) didn't accept DSA keys anymore. They're apparently not save enough anymore. I worked around it by adding a setting on the server, telling it to accept my DSA key. When the latest Apple OS update (Sierra) came out, I suddenly couldn't log in anywhere. Also git pull to github (I'm using ssh) stopped working. I found the cause by passing -vvvv to ssh: debug1: Next authentication method: publickey debug1: Trying private key: /Users/reinout/.ssh/id_rsa debug3: no such identity: /Users/reinout/.ssh/id_rsa: No such file or directory debug1: Trying private key: /Users/reinout/.ssh/id_ecdsa debug3: no such identity: /Users/reinout/.ssh/id_ecdsa: No such file or directory debug1: Trying private key: /Users/reinout/.ssh/id_ed25519 debug3: no such identity: /Users/reinout/.ssh/id_ed25519: No such file or directory debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password Oh.... Sierra's ssh client also doens't like DSA anymore. Ok... time to create a … -
Django under the hood: testing - Ana Balica
(One of my summaries of a talk at the 2016 django under the hood conference). A quick history of django's testing framework. Ticket #2333 got added to django's bug tracker before 1.0 was out: we want an integrated test framework ("Rails has it too"). A while later there was a test runner that looked at tests.py and models.py. models.py? Yes, as at that time doctests were still very popular and models were commonly tested with doctests. The rest was for the normal tests in tests.py. Django 1.1 added a build-in testclient for basic GET/PUT. Also TransactionTestCase was added: this one rolled back database transactions at the end of the tests. Better performance. 1.2 added a new class-based test runner. You could now also terminate the entire test run upon the first error ("failfast"). 1.3 splits the old test client into an actual client and a RequestFactory. Well, the client is a subclass of RequestFactory, something Ana doesn't like and would like to see refactored during the sprints. Doctests turned out not to be an ideal combination of tests and documentation. Testing was harder and the documentation not clear. So doctests were discouraged. In 1.4, more TestCases were added. SimpleTestcase for … -
Django under the hood: debugging performance - Aymeric Augustin
(One of my summaries of a talk at the 2016 django under the hood conference). Performance? Partially it is a question of perception. Up to 0.1s is "reacting instantaneously". Up to 1s: "not interrupting the user's flow of thought". Up to 10 seconds is slow, but the user might keep waiting. More than 10 seconds and they're off to check facebook. To optimize something, we have to measure it. For instance page load time. You could use your browser's development tools to see how quick a page loads. But you're measuring that on your fast development laptop. In chrome, you can make your internet connection worse and simulate 3G speed, for instance. Google analytics site speed can be used, it measures it in 1% of the cases a user looks at your page. Performance timeline Let's look at what happens for one single request. DNS lookup. This can take a surprising amount of time. Establish a TCP connection. Finally send over the request to the webserver. You receive the first byte of the response. After a while the last byte comes in. Page processing in the browser itself. Rendering the page and so. onLoad javascript and so. On average, the … -
Django Tips #19 Protecting Sensitive Information
The internet is a wild land. Security must be priority one when deploying a web application on the internet. The Django framework does an amazing job providing reliable and secure APIs. But none of that matters if we don’t use them properly. Nothing new that we should never deploy a Django application with DEBUG=True, right? One of the features of having DEBUG=True is dumping lots of metadata from your environment, including the whole settings.py configurations, when a exception occurs. Even though you will never be using DEBUG=True, you need extra care when naming the configurations in the settings.py module. Make sure all sensitive variables use one of the keywords: API KEY PASS SECRET SIGNATURE TOKEN This way, Django will not dump those variables that may contain sensitive information. Do S3_BUCKET_KEY = 'xxxxxxxxxxxxxxxx' Don't S3_BUCKET = 'xxxxxxxxxxxxxxxx' CHAVE_DE_ACESSO = 'xxxxxxxxxxxxxxxx' # "access key" in portuguese Even when you are running your application with DEBUG=False, if it’s configured to send error reports via email, there is a chance of the error report being exposed, specially if you are transmitting error reports unencrypted over the internet. PS: I mention this a lot here in the blog, but it’s never enough: Don’t commit sensitive … -
How to Handle GitHub Webhooks Using Django
Webhooks are a convenient way to notify external services when a certain event occur. GitHub provides an easy way to create Webhooks for the git repositories. You can pick the events such as push, pull requests, and only be notified when they occur. It can be used to integrate external applications with GitHub, perform Continuous Integration tasks or automate deployments. In this tutorial I will show you how to securely handle those notifications in a Django application. Creating the Webhook Let’s create a basic example so we can get more comfortable with the concept. First create a route to handle to Webhook POST data: urls.py: from core import views urlpatterns = [ ... url(r'^api/hello/$', views.hello, name='hello'), ] views.py from django.http import HttpResponse from django.views.decorators.csrf import csrf_exempt @csrf_exempt def hello(request): return HttpResponse('pong') Since by default Django protects views from receiving POST requests without the csrf middleware token, and in this case we are creating a view to actually handle a POST request from outside our application, we need to decorate the view function with the @csrf_exempt decorator. Now we go to GitHub and access the repository we want to create the Webhook. Go to Settings then Webhooks. Then inside Webhooks we … -
JSON Web Tokens in django application- part three
As we have working application now it's high time to make it more secure by authenticating users. To do this I will use JSON Web Tokens. Table of Contents: JWT in Django Rest Framework Implementing JWT in DRF application Other blog posts in this series JWT in Django Rest Framework There are few packages on pypi that provide JWT support but as I am already using DRF I choose package called REST framework JWT Auth. It's simple package and does it's job well so I can recommend it to everyone. But you have to make sure that your application is behind SSL/TLS as JWT tokens generated are not signed. But enough writing- let's jump into the code. Implementing JWT in DRF application First I added small change to my Task model definition in models.py: class Task(models.Model): # rest of model person = models.ForeignKey('auth.User', related_name='tasks') # rest of model It is the same model definition but written using string. The code in Django responsible for model lookup based on the string can be seen here. Then I added an additional field to UserSerializer- thanks to that when getting info about the user I also get info about which tasks this user … -
JSON Web Tokens in django application- part three
As we have working application now it's high time to make it more secure by authenticating users. To do this I will use JSON Web Tokens. Table of Contents: JWT in Django Rest Framework Implementing JWT in DRF application Other blog posts in this series JWT in Django Rest Framework … -
Django test optimization with no-op PIL engine
The Air Mozilla project is a regular Django webapp. It's reasonably big for a more or less one man project. It's ~200K lines of Python and ~100K lines of JavaScript. There are 816 "unit tests" at the time of writing. Most of them are kinda typical Django tests. Like: def test_some_feature(self): thing = MyModel.objects.create(key='value') url = reverse('namespace:name', args=(thing.id,)) response = self.client.get(url) .... Also, the site uses sorl.thumbnail to automatically generate thumbnails from uploaded images. It's a great library. However, when running tests, you almost never actually care about the image itself. Your eyes will never feast on them. All you care about is that there is an image, that it was resized and that nothing broke. You don't write tests that checks the new image dimensions of a generated thumbnail. If you need tests that go into that kind of detail, it best belongs somewhere else. So, I thought, why not fake ALL operations that are happening inside sorl.thumbnail to do with resizing and cropping images. Here's the changeset that does it. Note, that the trick is to override the default THUMBNAIL_ENGINE that sorl.thumbnail loads. It usually defaults to sorl.thumbnail.engines.pil_engine.Engine and I just wrote my own that does no-ops in … -
Handling statuses in Django #2
As discussed in the previous article, django-fsm is a great plugin to handle statuses in Django. Let's use it with an API. -
Handling statuses in Django #2
As discussed in the previous article, django-fsm is a great plugin to handle statuses in Django. Let's use it with an API. -
Understanding Checkout flow in Django Oscar.
If you are new to Django Oscar, then you can read these blog posts to understand how to create your own e-commerce shop using django-oscar and how to customize django oscar models, views and url's. Django Oscar Checkout Flow covers the most use-cases. You can skip steps you don't need, or just write your own checkout views. In Oscar the checkout process is laid out like this: Step 1 - Gateway * Logged in users will be automatically redirected to the next step. * Anonymous users are prompted to sign in, or to proceed as a guest. NOTE: Even as a guest user, oscar still collects their email address. Step 2 - Collect shipping information * Logged in users are offered the choice of either to enter or choose a shipping address from the existing user addresses. * Anonymous users are asked to enter shipping address details. Info - When the user enters a shipping address, then this address will be saved in the session and later saved as ShippingAddress model instance when the order is successfully submitted. Step 3 - Choose a shipping method * … -
Presidential debate questions influenced by open source platform
During the past two presidential debates, moderators from ABC and Fox asked candidates Hillary Clinton and Donald Trump voter-submitted questions from PresidentialOpenQuestions.com. The site, created by the bipartisan Open Debate Coalition (ODC), was built with the support of Caktus Group using an open source Django code base. “This coalition effort is a first-of-its-kind attempt to ensure moderators can ask questions that are not just submitted by the public, but voted on by the public to truly represent what Republican, Democratic, and Independent families are discussing around their dinner tables. Open Debates are the future,” said Lilia Tamm Dixon, Open Debate Coalition Director. Voters using PresidentialOpenQuestions.com submitted over 15,000 questions and cast more than 3.6 million votes for their favorite submissions. The selected debate questions had an unprecedented audience. According to Nielsen Media, 66.5 million viewers watched the second debate and 71.6 million the third debate. The ODC and Caktus teams continue to make improvements to the platform, readying new versions for use in political debates around the country. For national media coverage on the Open Debate Coalition and to learn more about their goals, see articles from The Atlantic, The Los Angeles Times, and Politico. -
ShipIt Day Recap Q3 2016
This ShipIt day marks four years of ShipIt days at Caktus! We had a wide range of projects that people came together to build. Most importantly, we all had fun and learned through actively working on the projects. People explored new technologies and tools, and had a chance to dig a bit deeper into items that piqued their interest in their regular work. React + Django = django-jsx Calvin did some work inspired by a client project to create tools for working with React’s JSX DOM manipulation within Django projects. This bridge allows embedding of JSX in Django templates (even using Django template language syntax) to be compiled and then rendered on the page. Calvin released django-jsx up on Github and pypi, and is interested in feedback from people who use it. Django ImageField compression Dmitriy continued working on the TinyPNG compressed Django ImageField from the previous ShipIt Day. He’s shared his updates through the Github repository django_tinypng. This time Dmitriy worked on cleaning up the project in preparation for possibly submitting it to pypi. His work included adding documentation and a nice way to migrate pre-existing image fields in projects to the new compressed image field. Python microservices with … -
How to Add Social Login to Django
In this tutorial we will implement Facebook, Twitter and GitHub authentication using the python-social-auth library. They support several other services and the process should be somewhat similar. The python-social-auth library have several customization options, which sometimes makes it challenging to get started. So for this tutorial I will guide you through the mandatory steps where in the end you will have a fully functional social authentication. Installation As usual, go for pip: pip install python-social-auth Add the social.apps.django_app.default to your INSTALLED_APPS: INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'social.apps.django_app.default', # <-- 'mysite.core', ] Migrate the database: python manage.py migrate Operations to perform: Apply all migrations: admin, auth, contenttypes, sessions, social_auth Running migrations: Applying social_auth.0001_initial... OK Applying social_auth.0002_add_related_name... OK Applying social_auth.0003_alter_email_max_length... OK Applying social_auth.0004_auto_20160423_0400... OK Applying social_auth.0005_auto_20160727_2333... OK The library will automatically handle authentication tokens and all the required information to deal with OAuth and OAuth2. Generally speaking, you won’t need to handle it manually nor access the user’s social profile. Configuration Update the MIDDLEWARE_CLASSES by adding the SocialAuthExceptionMiddleware to the end of it: MIDDLEWARE_CLASSES = [ 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.auth.middleware.SessionAuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'social.apps.django_app.middleware.SocialAuthExceptionMiddleware', # <-- ] Now we update the context_processors inside TEMPLATE: TEMPLATES = [ … -
JSON Web Tokens in django application- part two
In this blog post, I will deal with creating simple Django application for creating tasks using django rest framework. From this blog post, you can learn how to setup basic DRF application. Table of Contents: Overview of application Application code Other blog posts in this series Overview of application The main goal of this application is to create tasks. Each task has a title - string with a maximum length of 100 characters. Task also has a person to which it is bound (many to one relation - ForeginKey). The last thing that task have is date and time which given task is due to. The user can easily modify each of tasks so GET, POST, PUT and DELETE methods are supported. As we know how the application is designed let's jump into the code. Application code First, there is a need to create model for Task: from django.db import models from django.contrib.auth.models import User class Task(models.Model): title = models.CharField(max_length=100) person = models.ForeignKey(User) due_to = models.DateTimeField() def __str__(self): return 'Task with title: {}'.format(self.title) The arguments of Task correspond to what was written in the overview. As we have models ready now it's time to create serializers so data from database … -
JSON Web Tokens in django application- part two
In this blog post, I will deal with creating simple Django application for creating tasks using django rest framework. From this blog post, you can learn how to setup basic DRF application. Table of Contents: Overview of application Application code Other blog posts in this series Overview of application The …