Django community: Community blog posts RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Why using factories in Django
From the very beginning of a project, you need some data. You need data in your development database and you need data for your automated tests. The instinctive solution is to manually enter a set of data via the Django admin. The official way is to enter data via Django fixtures file(s). Using factories will make it easier and better; here is why. -
What We’re Clicking - April Link Roundup
It's time for this month’s roundup of articles and posts shared by Cakti that drew the most attention on Twitter. The list highlights new work in civic tech and international development as well as reasons for the increasing popularity of Python and open source development. Python is an Equal Opportunity Programming Language An interview with David Stewart, manager in the Intel Data Center Software Technology group, about the unique accessibility of the Python programming language as well as the inclusivity of its community. Why Every Developer is an Open Source Developer Now A short article on why the future of IT lies in open source collaboration. A Debate Where the Voters Pick the Questions The Atlantic’s profile of the Florida Open Debate platform. Caktus Group helped build the tool on behalf of the Open Debate Coalition. The platform powered the first-ever crowd sourced open Senate debate. Making it Easy to Bring Cellphone Apps to Africa A wonderful Fastco profile on Africa’s Talking, a startup devoted to making it easier for developers to disseminate SMS-based apps to cell phone users in Africa. -
How to track Google Analytics pageviews on non-web requests (with Python)
tl;dr; Use raven's ThreadedRequestsHTTPTransport transport class to send Google Analytics pageview trackings asynchronously to Google Analytics to collect pageviews that aren't actually browser pages. We have an API on our Django site that was not designed from the ground up. We had a bunch of internal endpoints that were used by the website. So we simply exposed those as API endpoints that anybody can query. All we did was wrap certain parts carefully as to not expose private stuff and we wrote a simple web page where you can see a list of all the endpoints and what parameters are needed. Later we added auth-by-token. Now the problem we have is that we don't know which endpoints people use and, as equally important, which ones people don't use. If we had more stats we'd be able to confidently deprecate some (for easier maintanenace) and optimize some (to avoid resource overuse). Our first attempt was to use statsd to collect metrics and display those with graphite. But it just didn't work out. There are just too many different "keys". Basically, each endpoint (aka URL, aka URI) is a key. And if you include the query string parameters, the number of keys … -
Florida Open Debate Platform Receives National Attention (The Atlantic, USA Today, Engadget)
Several national publications have featured the Florida Open Debate platform, including USA Today, Engadget, and The Atlantic. Caktus helped develop the Django-based platform on behalf of the Open Debate Coalition (ODC) in advance of the nation’s first-ever open Senate debate held in Florida on April 25th. The site enabled citizens to submit debate questions as well as vote on which questions mattered most to them. Moderators then used the thirty most popular questions from the site to structure the debate between Florida Senate candidates David Jolly (R) and Alan Grayson (D). According to The Atlantic,,more than 400,000 votes were submitted by users on the site, including more than 84,00 from Florida voters. “Normally, the press frames important US election debates by choosing the questions and controlling the video broadcast,” wrote Steve Dent. “For the first time, however, the public... decide[d] the agenda.” In his article for The Atlantic, Russell Berman also applauded the site’s effort “to make bottom-up, user-generated questions the centerpiece of a debate.” But possibly more significant were the results of this crowd-sourced content. “What transpired was, by all accounts, a decent debate,” Berman writes. “For 75 minutes, Grayson and Jolly addressed several weighty policy disputes—money in politics, … -
ES6 For Django Lovers
ES6 for Django Lovers! The Django community is not one to fall to bitrot. Django supports every new release of Python at an impressive pace. Active Django websites are commonly updated to new releases quickly and we take pride in providing stable, predictable upgrade paths. We should be as adamant about keeping up that pace with our frontends as we are with all the support Django and Python put into the backend. I think I can make the case that ES6 is both a part of that natural forward pace for us, and help you get started upgrading the frontend half of your projects today. The Case for ES6 As a Django developer and likely someone who prefers command lines, databases, and backends you might not be convinced that ES6 and other Javascript language changes matter much. If you enjoy the concise expressiveness of Python, then ES6's improvements over Javascript should matter a lot to you. If you appreciate the organization and structure Django's common layouts for projects and applications provides, then ES6's module and import system is something you'll want to take advantage of. If you benefit from the wide variety of third-party packages the Python Package index makes … -
Multi-table Inheritance and the Django Admin
Django's admin interface is a great way to be able to interact with your models without having to write any view code, and, within limits, it's useful in production too. However, it can quickly get very crowded when you register lots of models. Consider the situation where you are using Django's multi-table inheritance: {% highlight python %} from django.db import models from model_utils.managers import InheritanceManager class Sheep(models.Model): sheep_id = models.AutoField(primary_key=True) tag_id = models.CharField(max_length=32) date_of_birth = models.DateField() sire = models.ForeignKey('sheep.Ram', blank=True, null=True, related_name='progeny') dam = models.ForeignKey('sheep.Ewe', blank=True, null=True, related_name='progeny') objects = InheritanceManager() class Meta: verbose_name_plural = 'sheep' def __str__(self): return '{}: {}'.format(self._meta.verbose_name, self.tag_id) class Ram(Sheep): sheep = models.OneToOneField(parent_link=True) class Meta: verbose_name = 'ram' verbose_name_plural = 'rams' class Ewe(Sheep): sheep = models.OneToOneField(parent_link=True) class Meta: verbose_name = 'ewe' verbose_name_plural = 'ewes' {% endhighlight %} Ignore the fact there is no specialisation on those child models: in practice you'd normally have some. Also note that I've manually included the primary key, and the parent link fields. This has been done so that the actual columns in the database match, and in this case will all be `sheep_id`. This will make writing joins slightly simpler, and avoids the (not specific to Django) ORM anti-pattern of … -
(Directly) Testing Django Formsets
Django Forms are excellent: they offer a really nice API for validating user input. You can quite easily pass a dict of data instead of a `QueryDict`, which is what the request handling mechanism provides. This makes it trivial to write tests that exercise a given Form's validation directly. For instance: {% highlight python %} def test_my_form(self): form = MyForm({ 'foo': 'bar', 'baz': 'qux' }) self.assertFalse(form.is_valid()) self.assertTrue('foo' in form.errors) {% endhighlight %} Formsets are also really nice: they expose a neat way to update a group of homogenous objects. It's possible to pass a list of dicts to the formset for the `initial` argument, but, alas, you may not do the same for passing data. Instead, it needs to be structured as the `QueryDict` would be: {% highlight python %} def test_my_formset(self): formset = MyFormSet({ 'formset-INITIAL_FORMS': '0', 'formset-TOTAL_FORMS': '2', 'formset-0-foo': 'bar1', 'formset-0-baz': 'qux1', 'formset-1-foo': 'spam', 'formset-1-baz': 'eggs' }) self.assertTrue(formset.is_valid()) {% endhighlight %} This is fine if you only have a couple of forms in your formset, but it's a bit tiresome to have to put all of the prefixes, and is far noisier. Here's a nice little helper, that takes a `FormSet` class, and a list (of dicts), and instantiates … -
2016 DBIR Highlights
The 2016 edition of Verizon’s Data Breach Investigations Report is out, and as usual it’s compelling reading. The DBIR is one of the only sources of hard data about information security, which makes it a must-read for anyone trying to run a security program in a data-driven manner. What follows are the bits that I found especially interesting, and a bit of my own commentary. Internal threats are rare [T]he Actors in breaches are predominantly external. -
Florida Open Debate Site Powers First-Ever Crowd-Sourced Open Senate Debate
Florida Open Debate launched ahead of the upcoming, bi-partisan debate between candidates for the Florida Senate. The site, which crowdsources debate questions from the general public, was met with national acclaim. Citizens can not only submit questions, but also vote on which ones matter most. Caktus helped develop the tool on behalf of the Open Debate Coalition (ODC), a non-partisan organization dedicated to supporting participatory democracy through the use of civic tech. The coalition formed during the 2008 presidential election to combat a sharp decline in voter participation as well as lack of representation in the debate arena. According to The Los Angeles Times “the job of designing and choosing questions is left to the media host.” The ODC recognized the need to increase participation from and provide access to the debate system by reaching as many American citizens as possible. "The tool truly is an open forum for US citizens to participate in the political process,” says Ben Riesling, Caktus project manager. “Anyone can submit a question and vote on which questions should or should not be discussed. We’re extremely honored to be asked to participate in making this site available during this election season.” The debate between Florida … -
Django: Running management commands inside a Docker container
Okay, so we have dockerized our django app and we need to run a manage.py command for some task. How do we do that? Simple, we have to locate the container that runs the django app, login and then run the command. Locate The Container It’s very likely that our app uses multiple containers to compose the entire system. For exmaple, I have one container running MySQL, one container running Redis and another running the actual Django app. If we want to run manage.py commands, we have to login to the one that runs Django. While our app is running, we can find the running docker containers using the docker ps command like this: $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 308f40bba888 crawler_testscript "/sbin/my_init" 31 hours ago Up 3 seconds 5000/tcp crawler_testscript_1 3a5ccc872215 crawler_web "bash run_web.sh" 31 hours ago Up 4 seconds 0.0.0.0:8000->8000/tcp crawler_web_1 14f0e260fb2c redis:latest "/entrypoint.sh redis" 31 hours ago Up 4 seconds 0.0.0.0:6379->6379/tcp crawler_redis_1 252a7092870d mysql:latest "/entrypoint.sh mysql" 31 hours ago Up 4 seconds 0.0.0.0:3306->3306/tcp crawler_mysql_1 In my case, I am using Docker Compose and I know my Django app runs using the crawler_web image. So we note the name of the container. In … -
Autocomplete with Django-Haystack and Elasticsearch with single letter querying.
Django's haystack provides autocomplete functionality. To do autocomplete effectively, the search backend(elasticsearch in this case) uses n-grams (essentially a small window passed over the string). Because this alters the way your data needs to be stored. We had two choices: NgramField and EdgeNgramField used as n-grams in search backend. The major drawback of the n-grams is that they take minimum of 3 letters in the search query. Still EdgeNgramField or NgramField fields won't produce consistent results(For Eg: . By customising the Haystack's built in elastcisearch engine backend we can achieve single letter query with Charfield itself. In your search_indexes.py from haystack import indexes from myapp.models import Book class BookIndex(indexes.SearchIndex, indexes.Indexable): text = indexes.CharField(document=True, use_template=True) title = indexes.CharField(model_attr='title') pub_date = indexes.DateTimeField(model_attr='pub_date') def get_model(self): return Book Create backends.py in your app. And this file will contain the actual customized elasticsearch search engine backend. from django.conf import settings from haystack.backends.elasticsearch_backend import ElasticsearchSearchBackend from haystack.backends.elasticsearch_backend import ElasticsearchSearchEngine from haystack.backends.elasticsearch_backend import ElasticsearchSearchQuery from haystack.constants import VALID_FILTERS, FILTER_SEPARATOR, DEFAULT_ALIAS from haystack.inputs import Clean, Exact, PythonData, Raw from django.utils import six class CustomElasticsearchSearchQuery(ElasticsearchSearchQuery): def __init__(self, using=DEFAULT_ALIAS): super(CustomElasticsearchSearchQuery, self).__init__(using=DEFAULT_ALIAS) def build_query_fragment(self, field, filter_type, value): from haystack import connections query_frag = '' if not hasattr(value, 'input_type_name'): # Handle when we've got a ``ValuesListQuerySet``... if … -
How to create Custom User Model in Django?
Django provides built-in authentication which is good for most of the cases, but you may have needs that are being served by the existing system. For Ex: You want 'email' for authentication purpose rather than Django's username field and you want an extra field called 'display_name' as a full name for the logged in User. To meet the above requirements, we need to customize Django's built-in user model or substitute a completely customized model. In this blog post, we'll learn how to customize Django's built-in User model. Its good idea to extend Django User Model rather than writing whole new user model and there are a lot of ways but one good way is to extend it from Django itself. We get all features and properties of Django User and our own custom features on top of it. Add the following to your app that holds custom user model. In models.py from django.db import models from django.contrib.auth.models import AbstractUser class User(AbstractUser): name = models.CharField(max_length=100, blank=True, null=True) and specify your custom user model in settings.py AUTH_USER_MODEL = ‘your_app.User' If you want to use your custom user model in models.py, you can get that model with `settings.AUTH_USER_MODEL` or `get_user_model()` of Django's … -
A reading list for InfoSec engineers
I’ve started a curated reading list for InfoSec engineers. I was inspired by Mark McGranaghan’s Services Engineering reading list. I really enjoy these kinds of personal, highly-curated reading lists, and for some time I’ve wanted to pull together one of my own. This is my list, not a definitive one — that is, these are resources I’ve found useful. As such it has some biases: It’s oriented towards providers of Software-, Platform-, and Infrastructure-as-a-Service. -
Django REST Framework: Remember to disable Web Browsable API in Production
So this is what happened – I built an url shortening service at work for internal use. It’s a very basic app – shortens urls and tracks clicks. Two models – URL and URLVisit. URL model contains the full url, slug for the short url, created time etc. URLVisit has information related to the click, like user IP, browser data, click time etc and a ForeignKey to URL as expected. Two different apps were using this service, one from me, another from a different team. So I kept the Web Browsable API so the developers from other teams can try it out easily and they were very happy about it. The only job of this app was url shortening so I didn’t bother building a different home page. When people requested the / page on the domain, I would redirect them directly to /api/. Things were going really great initially. There was not very heavy load on the service. Roughly 50-100 requests per second. I would call that minimal load. The server also had decent hardware and was running on an EC2 instance from AWS. nginx was on the front while the app was run with uwsgi. Everything was so … -
Psychological safety in the InfoSec industry
My co-worker Eric Mill recently brought up the topic of psychological safety. Referencing a study by Google that points to psychological safety as a key factor in successful teams, Eric wrote: Maybe these situations sounds familiar to others (they definitely both are to me): Did you feel like you could ask what the goal was without the risk of sounding like you’re the only one out of the loop? -
Django Generic many to many field implementation
Django application has a GM2MField that combines the features of the standard Django "ManyToManyField" and "GenericForeignKey". Installation: pip install django-gm2m Django’s contenttype framework must be (django.contrib.contenttypes) mentionned in the INSTALLED_APPS. INSTALLED_APPS = [ ... 'django.contrib.contenttypes', ... 'gm2m', ] Uses Of django-gm2m: 1. reverse relations 2. prefetching 3. Allows you to customize the deletion behaviour Example Models: from django.db import models class Videos(models.Model): name = models.CharField(maxlength=100) class Articles(models.Model): name = models.CharField(maxlength=100) If you want to have a field for the preferred videos and articles of a User, you need to add GM2MField to the User model. from gm2m import GM2MField class User(models.Model): email = models.CharField(maxlength=100) username = models.CharField(maxlength=100) related = models.GM2MField() Now you can add videos, articles to the related set. Creating User object: user = User.objects.create(email="mp@mp.com", username="mp@mp.com") video = Videos.objects.create(name="video") article = Articles.objects.create(name="sample document") user.related.add(video) user.related.add(article) or you can add both videos, articles instances. user.related = [video, article] From User instance, you can fetch all related videos, articles set or you can filter model using the "Model" or "Model__in" keywords. list(user.related) # This provides list of model objects list(user.related.filter(Model=Videos)) # This … -
Factory Boy - An alternative for Fixtures
Factory Boy is a fixtures replacement tool. It allows you to use objects customized for the current test, while only declaring the test-specific fields. Setting up your factories: For testing the Django app, create a root directory named tests. Then create __init__.py, tests.py (to write your tests), factories.py (to write your model factories) in that directory. So, your app layout looks like: myapp |- __init__.py |- admin.py |- forms.py |- models.py |- tests |- __init__.py |- factories.py |- tests.py |- urls.py |- views.py Features of Factory Boy: It allows you to automate much of the testing scenarios. Some examples are: 1. Use a 'Sequence' object using a lambda function to dynamically create unique field values: EX: username = factory.Sequence(lambda n: 'user_%d' % n) 2. Use a LazyAttribute object to pick from a range of choices for a defined field EX: choice = factory.LazyAttribute(lambda s: random.choice(dict(MyModel.CHOICES).keys())) A basic factory: Factories declare a set of attributes used to instantiate an object. Here's a basic factory that … -
Celery Flower to monitor task queue
Celery is a task queue that is to built an asynchronous message passing system. It can be used as a bucket where programming tasks can be dumped. The program that passed the task can continue to execute and function responsively. To monitor the status of these celery tasks we use celery flower The Celery Flower is a tool for monitoring your celery tasks and workers. It's web based and allows you to see task progress, details, worker status. Install flower with pip pip install flower Install RabbitMQ Celery requires a messaging agent in order to handle requests from an external source. This agent is referred to as a "broker". RabbitMQ is the messaging server which powers this service. RabbitMQ does the following: Listens for messages sent via the AMQP protocol. Stores these messages in one or more queues. Releases these messages to workers for consumption and processing. RabbitMQ does not actually execute tasks. Rather, messages will be sent and stored in queue untill executed. Install rabbitmq using sudo apt-get install rabbitmq-server Add broker_api url in your settings.py RabbitMQ management api broker_api = 'http://guest:guest@localhost:15672/api/' # Enable debug logging logging = 'DEBUG' We need to configure RabbitMQ for message broker services before … -
How to Create initial django migrations for existing DB schema.
Django provides the comfort database migrations from its version 1.8, with which we can avoid the usage of third party packages like the south. Adding migrations to new apps is straightforward - they come preconfigured to accept migrations, and so just run make migrations once you’ve made some changes. But if your app already has models and database tables, and doesn’t have migrations yet (for example, you created it against a previous Django version) or you got your migrations messed up, you’ll need to convert your app to use migrations. Following are the steps to create initial migrations to your app: Step1: Empty the django_migrations table: Just go to your corresponding database terminals and delete all the records from you django_migrations table with delete from django_migrations; Step2: Remove all the files in migrations folders in each and every app of your project. Go to terminal and run remove all files in migrations folder with rm -rf <app>/migrations/ Step3: Reset the migrations for the "built-in" apps: Reset all the migrations of the Django's built-in apps like admin with the command python manage.py migrate --fake Step4: Create initial migrations for each and every app: For each app run: python manage.py makemigrations <app>. … -
Check test coverage in Django code with Coveralls
Coverage: It is a tool used for showing the percentage of your codebase covered by tests. Test Coverage is an important indicator of software quality and an essential part of software maintenance. It helps in evaluating the test cases by providing the data on different coverage items. It is a useful tool for finding untested part of a code. Test coverage can help in monitoring the quality of testing and assist in directing the test generators to create test cases that cover areas that have not been tested. It helps to measure the quality and identifies unnecessary test cases which will not increase coverage. Benefits of Test Coverage: Defect prevention in the project. It creates additional test cases to increase coverage. It helps in determining a quantitative measure of code coverage, which indirectly measures the quality of the application or product. Installation: Pip install coverage Use "coverage run" to run your program and gather data: coverage run manage.py test coverage report -m # provides the report for the tests Individual app test can be run using the command coverage run --source=app1,app2 manage.py test -
From Intern to Professional Developer: Advice on a Mid-Career Pivot
A few weeks ago, Rebecca Conley attended DjangoCon Europe 2016 in Budapest, Hungary. The event is a five-day conference that brings together Django lovers from all over the world to learn about and share each other’s experiences with Django. Rebecca gave an inspiring talk on her transition into web development from other fields, including the non-profit sector. She approached the topic from a unique diversity in tech perspective, arguing that developers making such transitions have a great deal to offer the developer community as a whole. You can watch her talk below or check out the many other great talks here. -
Adopting Scrum in a Client-services, Multi-project Organization
Caktus began the process of adopting Scrum mid-November 2015 with two days of onsite Scrum training and fully transitioned to a Scrum environment in January 2016. From our original epiphany of “Yes! We want Scrum!” to the beginning of our first sprint, it took us six weeks to design and execute a process and transition plan. This is how we did it: Step 1: Form a committee Caktus is a fairly flat organization and we prefer to involve as many people as possible in decisions that affect the whole team. We formed a committee that included our founders, senior developers, and project managers to think through this change. In order for us to proceed with any of the following steps, all committee members had to be in agreement. When we encountered disagreement, we continued communicating in order to identify and resolve points of contention. Step 2: Identify an approach Originally we planned to adopt Scrum on a per-project basis. After all, most of the literature on Scrum is geared towards projects. Once we started planning this approach, however, we realized the overhead and duplication of effort required to adopt Scrum on even four concurrent projects (e.g. requiring team members to … -
Basics of Django messages framework
In any web application, we need to display notification messages to the end user after processing a form or some other types of his requests. To make this messaging system simple, Django provided full support to cookie and session based messaging for both anonymous and authenticated users. This messages framework is capable of storing messages in one request and retrieving those messages in the subsequent request. Every message has a tag based on its priority(info,warning and error). Enabling Django messages: -> we need to put 'django.contrib.messages' in INSTALLED_APPS. -> MIDDLEWARE_CLASSES should contains 'django.contrib.sessions.middleware.SessionMiddleware' and 'django.contrib.messages.middleware.MessageMiddleware'. -> The 'context_processors' option of the DjangoTemplates backend defined in your TEMPLATES setting contains 'django.contrib.messages.context_processors.messages'. storage.fallback.FallbackStorage is the default storage class.if you want, you can select another storage class by setting MESSAGE_STORAGE to its full import path, for example MESSAGE_STORAGE = 'django.contrib.messages.storage.cookie.CookieStorage' To write your own storage class, subclass the BaseStorage class in django.contrib.messages.storage.base and implement the _get and _store methods. Add messages: add_message(request, level, message, extra_tags='', fail_silently=False) from django.contrib import messages messages.add_message(request, messages.INFO, 'Hello world.') Some shortcut methods to add messages with commonly used tags. messages.debug(request, '%s SQL statements were executed.' % count) messages.info(request, 'Three credits remain in your account.') messages.success(request, 'Profile details updated.') messages.warning(request, 'Your account … -
Basics of Django templates
Django template engine comes as part of django framework and its very easy, extensible and handy to develop complex web applications with simplest efforts. Lets see basics of django template engine. Basics: A template is a text document, or a normal Python string, that is marked-up using the Django template language. Template system in Python can be used in the following way: First, compile the raw template code into a Template object. Then, call the render() method of the Template object with a given context. Rendering a context: Once you have a compiled Template object, you can render a context or multiple contexts with it. We have a Context class at django.template.Context, and its constructor takes two (optional) arguments: 1. A dictionary mapping variable names to variable values. 2. The name of the current application. Example: from django.template import Template, Context # Creating a Template object by instantiating it and its constructor takes one argument i.e., raw template code. template = Template("My name is {{my_name}}") context = Context({"my_name":"XXX"}) # Calling the Template object's render() method with the context to fill the template. template.render(context) Templates: A template is simply a text file. It can generate any text-based … -
Add captcha to django web page using Python-reCaptcha
Python-reCaptcha is a pythonic and well-documented reCAPTCHA client that supports all the features of the remote API to generate and verify CAPTCHA challenges. To add into your django project, you need captcha public, private keys are required. How to integrate reCaptcha with django? Following gives you a brief idea of how to integrate reCaptcha to Django application First get the corresponding public key and private key from reCaptcha site and keep them in your settings.py in settings.py CAPTCHA_PUBLIC = 'XXXXXXXXXXXXXXXXXX' CAPTCHA_PRIVATE = 'XXXXXXXXXXXXXXXXXX' Render the reCaptcha to corresponding html page in views.py recaptcha = captcha.displayhtml(CAPTCHA_PUBLIC) #displayhtml() gives the captcha object to be displayed on html return render_to_response('demo.html','recaptcha':recaptcha) in demo.html <form action="/" method="post"> {{recaptcha|safe}} <button class="submit" type="submit">Submit</button> </form> Validate the challenge field and response fields.. in views.py capresponse = captcha.submit( request.POST.get('recaptcha_challenge_field'), request.POST.get('recaptcha_response_field'), CAPTCHA_PRIVATE, request.META.get('REMOTE_ADDR')) '''checks if the catpcha is valid or not(challenge field and response fields are equal or not ''' if capresponse.is_valid: do_some_work() else: return HttpResponse('wrong captcha text')