Django community: Community blog posts RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Django: From Runserver to Reddit Hugs
Last month, I presented High Performance Django: From Runserver to Reddit Hugs at DjangoCon US in Portland. My assertion was that Django, left to its own devices, does not scale. With the right supporting servers, however, it can scale fantastically. I gave a live demo of a Django site in multiple different server configurations with Docker on EC2, showing how each one affected performance. For those who missed the conference, here's the video: -
Read-only data from your database in Django
I had the need to create a column in some database tables that are completely controlled by the database, but the value of which is _sometimes_ needed by the [Django](https://www.djangoproject.com/) object. It should never be presented in a Form, and never, ever be written to by the Django infrastructure. So, we need a way to fetch the data from the database, but, even if the value is changed, and the object saved, is not written back. The detail of how this data is set in the database is irrelevant: it's a column that gets it's value from a sequence (and, incidentally, this sequence is shared across multiple tables). But, we need a way to get this data. A nice technique is to leverage two parts of Django: the `QuerySet.extra(select={})` method to actually add this field to the query, and `Manager.get_query_set()` (`get_queryset()` in older versions of Django) to make this apply to every fetch of the objects. Our extra column will be called `sequence_number` {% highlight python %} class SequenceNumberManager(models.managers.Manager): def get_query_set(self): return super(SequenceNumberManager, self).get_query_set().extra(select={ 'sequence_number': '"%s"."sequence_number"' % self.model._meta.db_table }) class Thing(models.Model): # Column definitions. Do not define sequence_number! objects = SequenceNumberManager() {% endhighlight %} That's it. Now, `Thing.objects.all()[0].sequence_number` will give … -
The Class Based "View"
People often find working with class based views hard, but they are simple... Once you spend time figuring them out. In this video start with the base of building blocks and work your way through completly understanding the base View of (generic) class based views.Watch Now... -
New Django Server Setup: Part 2
New Django Server Setup: Part 2 -
Celery in Production
(Thanks to Mark Lavin for significant contributions to this post.) In a previous post, we introduced using Celery to schedule tasks. In this post, we address things you might need to consider when planning how to deploy Celery in production. At Caktus, we've made use of Celery in a number of projects ranging from simple tasks to send emails or create image thumbnails out of band to complex workflows to catalog and process large (10+ Gb) files for encryption and remote archival and retrieval. Celery has a number of advanced features (task chains, task routing, auto-scaling) to fit most task workflow needs. Simple Setup A simple Celery stack would contain a single queue and a single worker which processes all of the tasks as well as schedules any periodic tasks. Running the worker would be done with python manage.py celery worker -B This is assuming using the django-celery integration, but there are plenty of docs on running the worker (locally as well as daemonized). We typically use supervisord, for which there is an example configuration, but init.d, upstart, runit, or god are all viable alternatives. The -B option runs the scheduler for any periodic tasks. It can also be run … -
How we use Virtualenv, Buildout and Docker
There are several technologies (in the Python world) to have isolated environments for projects. In this article I will describe how we use Virtualenv, Buildout and Docker for a project I’m working on at Fox-IT. Virtualenv The first tool I’ll discuss here is Virtualenv. According to its documentation Virtualenv is a tool to create isolated Python environments. What does it do? It offers a way to install Python packages independent of the global site-packages directory. This provides you with a way to install packages even when you do not have permission to write in the global site-packages directory and it will prevent conflicts with packages installed there (or in other Virtualenv environments for that matter). For instance on my current Ubuntu 14.04 installation has the requests package globally installed. However, it is version 2.2.1. What if I need a newer version? Or worse: what if my code is incompatible with a newer version and the package is updated for some reason (perhaps with a system upgrade)? How do we use it? For the project I’m working on, we have a couple of small tools written in Python that we need running in their own separate environment (on different machines than the code of the … -
How we use Virtualenv, Buildout and Docker
There are several technologies (in the Python world) to have isolated environments for projects. In this post I will describe how we use Virtualenv, Buildout and Docker for a project I’m working on at Fox-IT. Virtualenv The first tool I’ll discuss here is Virtualenv. According to its documentation Virtualenv is a tool to create isolated Python environments. What does it do? It offers a way to install Python packages independent of the global site-packages directory. This provides you with a way to install packages even when you do not have permission to write in the global site-packages directory and it will prevent conflicts with packages installed there (or in other Virtualenv environments for that matter). For instance my current Ubuntu 14.04 installation has the requests package globally installed. However, it is version 2.2.1. What if I need a newer version? Or worse: what if my code is incompatible with a newer version and the package is updated for some reason (perhaps with a system upgrade)? How do we use it? For the project I’m working on, we have a couple of small tools written in Python that we need running in their own separate environment (on different machines than the … -
Using Postgres Composite Types in Django
Note: this post turned out to be far more complicated than I had hoped. I may write another one that deals with a less complicated type! [Postgres](http://www.postgresql.org) comes with a pretty large range of column types, and the ability to use these types in an ARRAY. There's also [JSON(B)](http://www.postgresql.org/docs/9.4/static/datatype-json.html) and [Hstore](http://www.postgresql.org/docs/9.4/static/hstore.html), which are useful for storing structured (but possibly varying) data. Additionally, there are also a range of, well, [`range`](http://www.postgresql.org/docs/9.4/static/rangetypes.html) types. However, sometimes you actually want to store data in a strict column, but that isn't a simple scalar type, or one of the standard range types. Postgres allows you to define your own composite types. There is a command [`CREATE TYPE`](http://www.postgresql.org/docs/9.2/static/sql-createtype.html) that can be used to create an arbitrary type. There are four forms: for now we will just look at Composite Types. We will create a Composite type that represents the opening hours for a store, or more specifically, the default opening hours. For instance, a store may have the following default opening hours: {% highlight text %} +------------+--------+---------+ | Day | Open | Close | +------------+--------+---------+ | Monday | 9 am | 5 pm | | Tuesday | 9 am | 5 pm | | Wednesday | … -
Python Programmers Support the Ada Initiative
Please join me in helping making the Python community better by supporting the Ada Initiative. Last year, the Python community raised $10,000 for the Ada Initiative. This year I am betting we can beat that record and raise $20,000! Carl Meyer, Alex Gaynor, Jim Meyer, and I are pledging to match up to $10,000 in donations from the Python community; please join us in donating! Update: Minutes after we annonunced this, Jim Meyer offered an additional $2,500, bringing the match total to $10,000! -
On Flying Starts
What a start for a blog to end up on the homepage of Hacker News on the first day (with a <a title="Django UI in 2005 vs Django UI in 2014" href="/en/hoyci/2014/09/django-2005-2014/">short post about Django UI</a>). The good: it's surprisingly easy to install a cache plugin to Wordpress. The bad: this blog is not hosted on the Slipmat.io server (because I don't want to install PHP on it) so I missed a great chance to stress-test it.<img class="aligncenter size-full" src="https://www.unessa.net/static/kuvat/hoyci/screen-shot-2014-09-18-at-183557.png" alt="254 Concurrent Users in Google Realtime Analytics" /> To stay with the topic on flying starts, I've been quietly readying the first Slipmat.io tool, a site for taking and making requests in Mixify events, for its official launch. I opened the site for public three weeks ago and I've been working on it ever since. I'm keeping <a href="http://slipmat.io/requests/about/release_notes/">detailed public release notes</a> to keep the users updated and also for myself as a way to get a feel of the progress. Most non-DJs and furthermore non-Mixify users probably have no idea what this tool actually does, so <strong>let me explain it quickly</strong>. <a href="http://www.mixify.com/">Mixify</a> is a Web site where DJs can play for a live audience by streaming sound (and … -
Ramblings about data-structures
I was reading this the other day. The article presents this peculiar quote: "It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures." —Alan J. Perlis [1] And then some thoughts and questions regarding the quote that seemed a bit ridiculous. But what does this seemingly ambiguous quote really mean ? After some thought, I've concluded that it's an apologia for Lisp's lists everywhere philosophy. You can reduce the quote to: It's better to have one extremely generic type than 10 incompatible types. Python already does this: every object implements, surprise, an object interface that boils down to a bunch of magic methods. On the other hand, Lisp has generic functions. What this mean is that there is a dispatch system that binds certain function implementations to certain data-structures. A data structure that is bound with specific actions (aka functions or methods) is what I call a type. The idea of not having specialised data-structures is an illusion - if a function takes something as an input then you have assumed a specific data structure, not just a mere list. This is why I think it's worthwhile designing the … -
Ramblings about data-structures
I was reading this the other day. The article presents this peculiar quote: "It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures." —Alan J. Perlis [1] And then some thoughts and questions regarding the quote that seemed a bit ridiculous. But what does this seemingly ambiguous quote really mean? After some thought, I've concluded that it's an apologia for Lisp's lists everywhere philosophy. You can reduce the quote to: It's better to have one extremely generic type than 10 incompatible types. Python already does this: every object implements, surprise, an object interface that boils down to a bunch of magic methods. On the other hand, Lisp has generic functions. What this mean is that there is a dispatch system that binds certain function implementations to certain data-structures. A data structure that is bound with specific actions (aka functions or methods) is what I call a type. The idea of not having specialised data-structures is an illusion - if a function takes something as an input then you have assumed a specific data structure, not just a mere list. This is why I think it's worthwhile designing the data-structures … -
Ramblings about data-structures
I was reading this the other day. The article presents this peculiar quote: "It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures." —Alan J. Perlis [1] And then some thoughts and questions regarding the quote that seemed a bit ridiculous. But what does this seemingly ambiguous quote really mean? After some thought, I've concluded that it's an apologia for Lisp's lists everywhere philosophy. You can reduce the quote to: It's better to have one extremely generic type than 10 incompatible types. Python already does this: every object implements, surprise, an object interface that boils down to a bunch of magic methods. On the other hand, Lisp has generic functions. What this mean is that there is a dispatch system that binds certain function implementations to certain data-structures. A data structure that is bound with specific actions (aka functions or methods) is what I call a type. The idea of not having specialised data-structures is an illusion - if a function takes something as an input then you have assumed a specific data structure, not just a mere list. This is why I think it's worthwhile designing the … -
Ramblings about data-structures
I was reading this the other day. The article presents this peculiar quote: "It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures." —Alan J. Perlis [1] And then some thoughts and questions regarding the quote that seemed a bit ridiculous. But what does this seemingly ambiguous quote really mean? After some thought, I've concluded that it's an apologia for Lisp's lists everywhere philosophy. You can reduce the quote to: It's better to have one extremely generic type than 10 incompatible types. Python already does this: every object implements, surprise, an object interface that boils down to a bunch of magic methods. On the other hand, Lisp has generic functions. What this mean is that there is a dispatch system that binds certain function implementations to certain data-structures. A data structure that is bound with specific actions (aka functions or methods) is what I call a type. The idea of not having specialised data-structures is an illusion - if a function takes something as an input then you have assumed a specific data structure, not just a mere list. This is why I think it's worthwhile designing the data-structures … -
Django 1.7 新数据迁移工具 (migrations) 的使用和如何从 South 升级转换
在1.6之前, Django只支持添加新的model到数据库, 而无法编辑或修改已经存在的model. 在当时, 这些Django缺失的功能可以通过South实现. 1. 新的命令 Django 1.7 为我们带来了三个新命令: migrate: 用于执行迁移动作 makemigrations: 基于当前的model创建新的迁移策略文件 sqlmigrate: 显示迁移的SQL语句 值得注意的是, migration是基于App的, 因此, 我们可以针对某些app不启用migration功能. 2. 如何使用 migrations的使用非常简单: 修改model, 比如增加field, 然后运行 python manager.py makemigrations 你的mmodel会被扫描, 然后与之前的版本作比较, 在app的migrations目录下生成本次迁移文件. 我们建议查看一下该迁移文件, 确保没有问题. 然后运行: python manager.py migrate migrate命令会进行比较, 并应用该迁移. 3. 从South到新的Django migrations 如果想从south升级到最新的django migration, 可以按以下步骤实现: 确保south中的migration全部被应用了 从 INSTALLED_APPS中移除south 删除每个app下migration目录中的所有文件, 除了__init__.py 运行python manager.py makemigrations, Django会初始化migration 运行python manager.py migrate, django会发现数据库和初始化的migration相同, 从而将他们标记为已应用 -
Django UI in 2005 vs Django UI in 2014
I stumbled across my favourite ever tech talk, <a href="https://www.youtube.com/watch?v=cb9KDt9aXc8">the legendary Snakes and Rubies event</a> where Django met Rails, back in 2005. One thing caught my eye while re-watching every minute of the 3-hour event: the Django Web site and the Admin Site app look almost <em>exactly</em> the same today than they did 9 years ago. <img class="aligncenter size-full" src="/static/kuvat/hoyci/screen-shot-2014-09-17-at-225740.png" alt="Screen capture image from Djangoproject.com as it was in 2005" /> <p style="text-align: center;"><em>Still image of Djangoproject.com from the Snakes and Rubies video, 2005.</em></p> <p>&nbsp;<p> <p style="text-align: center;"><img class="aligncenter size-full" src="/static/kuvat/hoyci/screen-shot-2014-09-17-at-225028-1024x559.png" alt="Screen Shot 2014-09-17 at 22.50.28" /></p> <p style="text-align: center;"><em>Djangoproject.com, 2014.</em></p> <p>&nbsp;<p> <p style="text-align: center;"><img class="aligncenter size-full" src="/static/kuvat/hoyci/screen-shot-2014-09-17-at-225247.png" alt="Still image from Snakes and Rubies video" /></p> <p style="text-align: center;"><em>Still image of Django Admin Site from the Snakes and Rubies video, 2005.</em></p> <p>&nbsp;<p> <p style="text-align: center;"><img class="aligncenter size-full" src="/static/kuvat/hoyci/screen-shot-2014-09-17-at-225844.png" alt="Screen Shot 2014-09-17 at 22.58.44" /></p> <p style="text-align: center;"><em>A screencapture of one Django Admin app (in Finnish -- it still talks also Welsh, among others), 2014.</em></p> <p>&nbsp;<p> <p>I don't want to comment this in any other way but to notice that it's quite amazing that Djangoproject.com still looks so good and I actually <em>like</em> using it every single day -- it works … -
My take on the "12 factor app"
http://12factor.net/ is often quoted as a standard shopping list if you want to get your deployment right. At least in the Python web world it is, it seems to me. I'm currently looking at the way we deploy stuff at our company (Nelen & Schuurmans). Partially by hand, lots via fabic, increasing use of ansible. And many infrastructure-parts like the main web proxy config essentially by hand, aided by scripts. Not everything in the 12 factor app list is needed for us, but it helps me think about what we need to keep and what we need to improve. One codebase tracked in revision control, many deploys. We use git/github well. We also have multiple deploys, this works OK. Explicitly declare and isolate dependencies. Python packages and buildout. Pinning. Works fine. A few projects are less tidy, though, with git branch checkouts instead of tidy packages. Take care with javascript/css dependencies. The recent trend towards grunt, bower and so with a nice requirements file: looks good. On the whole, the way we compose the actual project works fine. Store config in the environment. DATABASES out of settings.py. We do this wrong now. Environment settings or, perhaps better, configuration in an … -
Adding Django form instance attributes
Sometimes in the clean(), clean_FOO or save() methods of a Django form, you need to have additional form instance attributes available. A sample case for this is having user_id available. This is a simple example of how to do it in Class-Based Views. Assuming this form: from django import forms from .models import MyModel class MyForm(forms.ModelForm): class Meta: model = MyModel def __init__(self, user_id, *args, **kwargs): super(MyForm, self).__init__(*args, **kwargs) # set the user_id as an attribute of the form self.user_id = user_id Now that the form is defined, the view needs to inject the form with the user id: from django.views.generic import UpdateView # this assumes that django-braces is installed from braces.views import LoginRequiredMixin from .forms import MyForm from .models import MyModel class MyUpdateView(LoginRequiredMixin, UpdateView): model = MyModel form_class = MyForm success_url = "/someplace/" def get_form_kwargs(self): """This method is what injects forms with their keyword arguments.""" # grab the current set of form #kwargs kwargs = super(MyUpdateView, self).get_form_kwargs() # Update the kwargs with the user_id kwargs['user_id'] = self.request.user.pk return kwargs Additional Notes You can use this technique with: forms.Form forms.ModelForm CreateView FormView UpdateView As always, http://ccbv.co.uk is a great resource for deliving into Django forms. django-vanilla-views This should also work … -
Adding Django form instance attributes
Sometimes in the clean(), clean_FOO or save() methods of a Django form, you need to have additional form instance attributes available. A sample case for this is having user_id available. This is a simple example of how to do it in Class-Based Views. Assuming this form: from django import forms from .models import MyModel class MyForm(forms.ModelForm): class Meta: model = MyModel def __init__(self, user_id, *args, **kwargs): super(MyForm, self).__init__(*args, **kwargs) # set the user_id as an attribute of the form self.user_id = user_id Now that the form is defined, the view needs to inject the form with the user id: from django.views.generic import UpdateView # this assumes that django-braces is installed from braces.views import LoginRequiredMixin from .forms import MyForm from .models import MyModel class MyUpdateView(LoginRequiredMixin, UpdateView): model = MyModel form_class = MyForm success_url = "/someplace/" def get_form_kwargs(self): """This method is what injects forms with their keyword arguments.""" # grab the current set of form #kwargs kwargs = super(MyUpdateView, self).get_form_kwargs() # Update the kwargs with the user_id kwargs['user_id'] = self.request.user.pk return kwargs Additional Notes You can use this technique with: forms.Form forms.ModelForm CreateView FormView UpdateView As always, http://ccbv.co.uk is a great resource for deliving into Django forms. While this technique is used … -
virtualenv Lives!
Setting up Python to the point to be able install packages from PyPI can be annoying and time-intensive. Even worse are OS-provided installations that start throwing cryptic error messages. Especially desktops are prone to that but it’s possible to break the whole toolchain of a server by installing some shiny package you heard about on reddit. Your desktop system is unlikely to be a throwaway virtual machine or container. Which makes it a highly mutable system with difficult rollbacks and a lot of pain if stuff breaks1. So until we all run NixOS on our desktops: Don’t install anything into its global site-packages beyond virtualenv. Does that sound extreme to you? Only if you haven’t found the right tools to make it effortless. virtualenv in 2014‽ virtualenv has been around for a while and was a somewhat accepted standard for installing Python software. Sadly, there are many missionaries running around nowadays, boldly proclaiming the end of virtualenv. Mostly because of containers in general and usually because of docker in particular. I find that unfortunate and shortsighted. Frankly, they fail to see the whole picture: virtualenv’s job isn’t just to separate your projects from each other. Its job is also to … -
virtualenv Lives!
Setting up Python to the point to be able install packages from PyPI can be annoying and time-intensive. Even worse are OS-provided installations that start throwing cryptic error messages. Especially desktops are prone to that but it’s possible to break the whole toolchain of a server by installing some shiny package you heard about on reddit. Your desktop system is unlikely to be a throwaway virtual machine or container. Which makes it a highly mutable system with difficult rollbacks and a lot of pain if stuff breaks. So until we all run NixOS on our desktops: Don’t pip-install anything into its global site-packages beyond virtualenv. Does that sound extreme to you1? Only if you haven’t found the right tools to make it effortless. virtualenv in 2014‽ virtualenv has been around for a while and was a somewhat accepted standard for installing Python software. Sadly, there are many missionaries running around nowadays, boldly proclaiming the end of virtualenv. Mostly because of containers in general and usually because of docker in particular. I find that unfortunate and shortsighted. Frankly, they fail to see the whole picture: virtualenv’s job isn’t just to separate your projects from each other. Its job is also to … -
Long Live Adjacency Lists
I recently wrote about the excellent book [SQL Antipatterns](http://pragprog.com/book/bksqla/sql-antipatterns), and in it briefly discussed the tree structures. I've been thinking about trees in [Postgres](http://www.postgresql.org/) a fair bit lately, and a discussion on [#django](https://botbot.me/freenode/django/) gave me further incentive to revisit this topic. The book discusses four methods of storing a tree in a database. *Adjacency Lists*, apart from the inability to grab a full or partial tree easily, are the simplest to understand. The child object stores a reference to it's parent. Because this is a foreign key, then it always maintains referential integrity. Fetching a parent is simple, as is fetching all children, or siblings. It's only when you need to fetch an arbitrary depth that things become problematic, unless you use a recursive query. More on that later. Postgres has an extension called [ltree](http://www.postgresql.org/docs/9.3/static/ltree.html), which provides an implementation of a *Path Enumeration*, but one thing that really bothers me about this type of structure is the lack of referential integrity. In practice, I'm not sure what having this `ltree` structure would give you over simply storing the keys in an `ARRAY` type. Indeed, if Postgres ever gets Foreign Key constraints for ARRAY elements (which there is a patch floating … -
Django-ckeditor-updated and django-ckeditor merge
I've released a new version of django-ckeditor-updated covering all latest changes (full ckeditor 4.4.4 package and some inline editor fixes). The django-ckeditor-updated-4.4.4 can be installed from pypi. Note that there is now a new required configuration variable - CKEDITOR_JQUERY_URL. All outstanding commits have been pushed to django-ckeditor repository (from which I forked) and released as django-ckeditor-4.4.4 (4.4.5 for updated readme). If you are using the "updated" version you can now switch to the django-ckeditor as I plan to make changes there as I now can push pypi releases for it. django-ckeditor-updated is closed. -
Announcing BarCamp Django SF!
BarCamp Django SF! On October 4th and 5th BarCamp Django SF will be taking place in the Eventbrite office in San Francisco. BarCamp Django SF is a low-cost, community-focused event that's different than any previous multi-day Django conference. So how is BarCamp Django SF different than previous Django conferences and events? Speakers are determined at the event I know what you are thinking. It's something on the line of "What the heck?!?" It means that if you want to give a talk, you don't need to be an expert. We don't have an formal, extensive review process. It's a chance to share ideas and learn something new. You can be just a Django hobbyist instead of a professional. All you have to be is passionate. At the event, we'll put up a board with slots for talks. Attendees can put themselves into these slots for talks. This makes for a fun, fluid event where new speakers are as welcome as experienced speakers. While this is meant to be an informal, beginner-friendly event, a professional level of conduct is still required: Speakers, like all attendees must agree to the code of conduct (also listed on the event page). No sales or … -
Announcing BarCamp Django SF!
BarCamp Django SF! On October 4th and 5th BarCamp Django SF will be taking place in the Eventbrite office in San Francisco. BarCamp Django SF is a low-cost, community-focused event that's different than any previous multi-day Django conference. How is BarCamp Django SF different than previous Django conferences and events? Speakers are determined at the event I know what you are thinking. It's something on the line of "What the heck?!?" It means that if you want to give a talk, you don't need to be an expert. We don't have an formal, extensive review process. It's a chance to share ideas and learn something new. You can be just a Django hobbyist instead of a professional. All you have to be is passionate. At the event, we'll put up a board with slots for talks. Attendees can put themselves into these slots for talks. This makes for a fun, fluid event where new speakers are as welcome as experienced speakers. While this is meant to be an informal, beginner-friendly event, a professional level of conduct is still required: Speakers, like all attendees must agree to the code of conduct (also listed on the event page). No sales or recruiting …