Django community: Community blog posts RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
PyCon.de: thursday lightning talks
(One of my summaries of a talk at the 2017 PyCon.de conference). (Note: if you have a name correction, mail me... Lightning talks go very fast) Note two: look at the mp3 player for very small kids talk below. That one was great. Work of a data analyst - Sofia Kosoran She talked yesterday about the difference between academia and industry. She now told an experience she had at her job. She didn't need deep learning or other hard stuff. Just a few minutes with pandas and taking the max meant that she became famous in her company :-) A linux firewall framework - Maximilian Wilhelm Alff: a linux firewall framework. For instance for a distributed linux firewall, based on a central point of truth/configuration. It knows about network topology, services and security classes. You can extend it with python plugins. If you don't know iptables: don't use this tool. See https://github.com/BarbarossaTM/alff Introducing BiCDaS - Nils Hachmeister BiCDaS: Bielefeld center for data science. It is a cross-section of different parts of the university, it is an initiative coming out of a round table. So: interdiciplinary. The participants come from all over the university: biology, sociology, law, technical, IT, etc. Something … -
PyCon.de: the python ecosystem for data science - Christian Staudt
(One of my summaries of a talk at the 2017 PyCon.de conference). Data science often has a similar workflow: acquire, ingest/clean, store/manage, data wrangling, visual analysis, modeling, story-telling. For many of those stages, python has nice tools. Christian Staudt calls it an ecosystem. Well, if you make a diagram showing the various tools it starts to look like one of those biology diagrams showing which kinds of animal eats what other kinds of animals. Likewise, python libraries have their function and their specialized niche. Data wrangling Numpy. The fundamental package for numeric computing in python. N-dimensional arrays. Numpy arrays are different to python lists: they're layed out in memory in a much more effective and compact way. Essential for understanding numpy: "lose your loops". Don't loop over arrays with regular python operations, but use numpy methods. That pushes everything down into highly effecient compiled code. That can gain you an order of magnitude in performance. Pandas. Labled indexed array data structures: series, dataframes, timeseries. It also include operations tailored to it like group_by and filter. And handy data import functionality (csv, excel, etc). He showed a quick example. It pays off to experiment with pandas and to explicitly Dask. Dask … -
PyCon.de keynote: Neutrinos - Susanne Mertens
(One of my summaries of a talk at the 2017 PyCon.de conference). Susanne Mertens is a physics professor in Munich. Neutrinos? Karlsruhe? What is the link? There is the KATRIN experiment with the 'KA' of Karlsruhe. An experiment looking more at neutrons. Computing and programming tools essential for scientific experiments. She'll tell about the physics background and end up with python. She showed some mind-boggling numbers. Huge number of galaxies. Galaxies made of huge numbers of stars. One of them being our sun. Our planet then has 7 billion people. Everyone exists out of a huge number of atoms. Multiply those numbers... But. All those atoms? 4% of the mass of the universe. The other 96%? Dark energy, dark matter. But what is it? There's lots of research into neutrinos. When you measure radioactive decay, you'd expect to measure a fixed amount of energy. But that's not the case. Someone postulated that maybe another unknown particle might be created: a neutrino. There are a lot of neutrinos. But: they hardly interact with regular matter. Enormeous numbers pass right through your body every second. So detecting and measuring them is a problem. It took 30 years to design an experiment to … -
PyCon.de: modern ETL-ing with Python and Airflow - Tamara Mendt
(One of my summaries of a talk at the 2017 PyCon.de conference). ETL? Extract, Transform, Load. It goes hand-in-hand with the traditional data warehousing. But that's the traditional sense. You can also see it as generic data transformation. The principles of ETL-ing are applicable. ETL often implies batch running. You could do it streaming, but most people still use batch processing. There are lots of commercial ETL tools. They have problems, however. They are mostly designed to deal with well structured data. They were made for moving from one DB system to another. They don't match well with the variety of sources you tend to have now. Mostly not open source You're limited to the build-in fuctionality. She suggest using python instead: Easy. More flexible. Reusing logic is easier. Abstraction is possible. You can test the logic. Versioning and collaboration. Airflow is a python platform to programmatically author, schedule and monitor workflows. It is a great tool to have the user-friendliness of commercial ETL tools and the flexibility of Python. It is python! It is open source. Note: it recently became an "apace incubator" project. There's a good community. Really nice: dynamic pieline and task creation. You can create them … -
PyCon.de: Python in space, the N body problem - Daniel Jilg
(One of my summaries of a talk at the 2017 PyCon.de conference). Daniel Jilg starts off with a picture of a celestial body (=sun, planets) movements with a geocentric worldview. When you put the earth in the center, the movements of the planets are hard to map. When you put the sun in the center, the movements make more sense. Circular movements or, as Keppler improved, elliptical movements. Newton improved on it with his law of gravity. Now the movements can be calculated. You can even look at the current planets' movements and look at the small incongruences. Would an extra planet, a long way off, fix those incongruences? Calculate it and start pointing your telescope. They're apparently in the pointing-your-telescope phase for some "planet nine" now. Once the number of celestial bodies gets higher, the calculus gets harder and takes longer. You can express that with the "big O notation". O: Order of magnitude. If you have a loop over all bodies inside a loop over all bodies.... you get n^2. That gets out of hand quite quickly. There's an algorithm that can do better. n log(n), which is much less worse than n^2. The algorithm places the bodies … -
PyCon.de: plugin ecosystems for Python web-applications - Raphael Michel
(One of my summaries of a talk at the 2017 PyCon.de conference). Raphael Michel asked for a show of hands. "Who is a python web developer". "Who used wordpress previously". Quite a number of us have worked with wordpress. About a quarter of the internet runs on wordpress or so. One of the reasons is its plugin mechanism that makes it possible to do lots of things. He works on pretix, a python ticket (concert tickets, not bug tracker tickets) system. They wanted to make it extensible, as there are many special cases (for instance payment processors). Their idea: Establish a way that plugins can hook into your applications. Provide many of these hooks themselves. Document it well They started with a simple signal system. A class that you can register functions on and a send method that simply calls all the registered functions. They set it up so that you could use decorators. If you want to use something like that and you're using django: use django's django.dispatch.signal. If you use django, do also package it up as a 'django applications'. But... if you want to use such an application, you have to: Install the package. Add the app … -
PyCon.de: wednesday lightning talks
(One of my summaries of a talk at the 2017 PyCon.de conference). Lightning talks, so I probably won't get all the names right. If you have additions/corrections, mail me :-) Automatic screenshots of your web app - Raphael Michel Raphael develops a web application. A web app needs documentation. Documentation needs screenshots. Making screenshots by hand is a lot of work. You have to do it over and over again for every new version. Selenium is a tool for instrumenting/steering your browser. He uses chrome --headless. So a proper browser, but without the GUI. py.test handles the various testcases, including the fixtures. Django has a LiveTestServerTestcase that is also handy. The screenshots are written like tests. Really simple and powerfull. Note: djangocon.eu 2018 will be in Heidelberg, Germany. 23-27 May. https://2018.djangocon.eu Counter-intuitive optimizations - Michael Penkov He had to make some counter-intuitive optimizations lately. He gathers data about websites and stores the info in mongodb. 500 million domains, more or less. He wanted, from an existing list, to know which domains weren't in the system yet. Doing it in python and querying the DB was slow. In the end he exported everything to text files and did it with linux … -
PyCon.de: vim your Python, Python your vim - Miroslav Šedivý
(One of my summaries of a talk at the 2017 PyCon.de conference). He has a nice simple keyboard layout. No weird key combinations. He started showing different keyboard layouts and started speaking fluently in en, de, sk, cs, fr, es, it, pl, sv, hu, eo, tr and explaining the various country's keyboard layouts. How do you manage all those languages one ONE keyboard with only ONE brain. Switching keyboards is no option. Charmaps are not easy. There used to be a key called the compose key. It was an actual key on older keyboards. You use compose + two or more keys after each other to get a é or an è, for instance. There's a x11 keymap for it. You can map various keys to function as the compose key. The printscreen key or the right control key or windows key, for instance. Another option is xcape, you can use one key (for instance the caps lock) as both the compose key and as another key. Keeping it pressed make it function as one, just pressing and releasing it quickly as the other. So: he uses the caps lock key as both ctrl and compose key. He then switched … -
PyCon.de: the borgbackup project - Thomas Waldmann
(One of my summaries of a talk at the 2017 PyCon.de conference). Borgbackup is 2.5 years old, but the code is older: is a fork of attic. Thomas discovered Attic after someone blogged about it. They forked it to get more collaboration and quicker releases. Borg backup is a backup tool. There are 1000 backup tools. So what's different? Borg is one you maybe actually would enjoy using. The features sound logical: simple, efficient, safe, secure. How borg sees this: Simple. Each backup is a full backup. Restore? Just do a FUSE mount. Easy pruning of old backups. Tooling: it is just borg, ssh and a shell. It is a single-file binary. There's good filesystem and OS support. There's good documentation. Efficient. It is very fast for unchanged files. Every backup is a full backup, but unchanged files don't need to be handled a second time. Chunk deduplication, sparse file support, flexible compression scheme. Compression is chunk-based, it doesn't compress the whole file at once. Safety. Checksums, transactions, filesystem syncing, atomic operations. Checkpoints while backing up. You can have off-site remote repositories. Secure. Authenticated encryption. There's nothing to see in the repo: borg doesn't trust the backup host, everything is … -
Five stories about the California Wildfires you probably missed
You’ve probably heard about the massive wildfires in Northern California. You probably know that they’re huge, that over 50 people have died, and that some wineries have burned. You might have seen some pictures. Unless you’ve been following closely, though, there’s a lot you’re missing. The vast majority of the reporting has lacked context, been overly sensationalistic, or has outright ignored deeper, more complex stories. It’s far deeper than a story about a natural disaster. -
How to Implement Django's Built In Password Management
Let's implement Django's built... -
Mercurial Mirror For Django 2.0 Branch
The first Beta was released today, so it seems a good day to start the mirror for the 2.0 branch of Django. For the record, main purposes of this mirror are: being a lightweight read-only repository to clone from for production servers hide the ugly git stuff behind a great mercurial interface The clone is […] -
Mailchimp Integration
The Mailchimp API is extensive... -
Automating Dokku Setup with AWS Managed Services
Dokku is a great little tool. It lets you set up your own virtual machine (VM) to facilitate quick and easy Heroku-like deployments through a git push command. Builds are fast, and updating environment variables is easy. The problem is that Dokku includes all of your services on a single instance. When you run your database on the Dokku instance, you risk losing it (and any data that's not yet backed up) should your VM suddenly fail. Enter Amazon Web Services (AWS). By creating your database via Amazon's Relational Database Service (RDS), you get the benefit of simple deploys along with the redundancy and automated failover that can be set up with RDS. AWS, of course, includes other managed services that might help reduce the need to configure and maintain extra services on your Dokku instance, such as ElastiCache and Elasticsearch. I've previously written about managing your AWS container infrastructure with Python and described a new project I'm working on called AWS Web Stacks. Sparked by some conversations with colleagues at the Caktus office, I began wondering if it would be possible to use a Dokku instance in place of Elastic Beanstalk (EB) or Elastic Container Service (ECS) to help … -
A Complete Beginner's Guide to Django - Part 7
Introduction Welcome to the last part of our tutorial series! In this tutorial, we are going to deploy our Django application to a production server. We are also going to configure an Email service and HTTPS certificates for our servers. At first, I thought about given an example using a Virtual Private Server (VPS), which is more generic and then using one Platform as a Service such as Heroku. But it was too much detail, so I ended up creating this tutorial focused on VPSs. Our project is live! If you want to check online before you go through the text, this is the application we are going to deploy: www.djangoboards.com. Version Control Version control is an extremely important topic in software development. Especially when working with teams and maintaining production code at the same time, several features are being developed in parallel. No matter if it’s a one developer project or a multiple developers project, every project should use version control. There are several options of version control systems out there. Perhaps because of the popularity of GitHub, Git become the de facto standard in version control. So if you are not familiar version control, Git is a good … -
PGDay.IT 2017
PGDay.IT is the Italian event dedicated to PostgreSQL, one of the most successful open-source projects. -
Kindle and ePub Now Available for Two Scoops of Django 1.11!
You asked for it, you got it. Two Scoops of Django 1.11 now comes in Kindle Mobi and ePub formats. If you've already purchased the PDF, you'll be receiving an email containing a new download link that adds the .mobi and .epub files. Otherwise, get yours at the following links: PDF/Kindle/ePub versions available here Amazon (Kindle only) -
Kindle and ePub Now Available for Two Scoops of Django 1.11!
You asked for it, you got it. Two Scoops of Django 1.11 now comes in Kindle Mobi and ePub formats. If you've already purchased the PDF, you'll be receiving an email containing a new download link that adds the .mobi and .epub files. Otherwise, get yours at the following links: PDF/Kindle/ePub versions available here Amazon (Kindle only) -
Kindle and ePub Now Available for Two Scoops of Django 1.11!
You asked for it, you got it. Two Scoops of Django 1.11 now comes in Kindle Mobi and ePub formats. If you've already purchased the PDF, you'll be receiving an email containing a new download link that adds the .mobi and .epub files. Otherwise, get yours at the following links: PDF/Kindle/ePub versions available here Amazon (Kindle only) -
Kindle and ePub Now Available for Two Scoops of Django 1.11!
You asked for it, you got it. Two Scoops of Django 1.11 now comes in Kindle Mobi and ePub formats. If you've already purchased the PDF, you'll be receiving an email containing a new download link that adds the .mobi and .epub files. Otherwise, get yours at the following links: PDF/Kindle/ePub versions available here Amazon (Kindle only) -
Simple or fancy UPSERT in PostgreSQL with Django
As of PostgreSQL 9.5 we have UPSERT support. Technically, it's ON CONFLICT, but it's basically a way to execute an UPDATE statement in case the INSERT triggers a conflict on some column value. By the way, here's a great blog post that demonstrates how to use ON CONFLICT. In this Django app I have a model that has a field called hash which has a unique=True index on it. What I want to do is either insert a row, or if the hash is already in there, it should increment the count and the modified_at timestamp instead. The Code(s) Here's the basic version in "pure Django ORM": if MissingSymbol.objects.filter(hash=hash_).exists(): MissingSymbol.objects.filter(hash=hash_).update( count=F('count') + 1, modified_at=timezone.now() ) else: MissingSymbol.objects.create( hash=hash_, symbol=symbol, debugid=debugid, filename=filename, code_file=code_file or None, code_id=code_id or None, ) Here's that same code rewritten in "pure SQL": from django.db import connection with connection.cursor() as cursor: cursor.execute(""" INSERT INTO download_missingsymbol ( hash, symbol, debugid, filename, code_file, code_id, count, created_at, modified_at ) VALUES ( %s, %s, %s, %s, %s, %s, 1, CLOCK_TIMESTAMP(), CLOCK_TIMESTAMP() ) ON CONFLICT (hash) DO UPDATE SET count = download_missingsymbol.count + 1, modified_at = CLOCK_TIMESTAMP() WHERE download_missingsymbol.hash = %s """, [ hash_, symbol, debugid, filename, code_file or None, code_id or … -
Disabling Error Emails in Django
One of Django's nice "batteries included" features is the ability to send emails when an error is encountered. This is a great feature for small sites where minor problems would otherwise go unnoticed. Once your site start getting lots of traffic, however, the feature turns into a liability. An error might fire off thousands of emails in rapid succession. Not only will this put extra load on your web servers, but you could also take down (or get banned from) your email server in the process. One of the first things you want to do when setting up a high-traffic Django site is replace the default error email functionality with an error reporting service like Sentry. Once you've got Sentry setup, what's the best way to disable error emails? Unfortunately, there isn't one simple answer. Let's dig into why... How does Django Send the Emails? First let's look at how Django sends those emails. If you look at the source, you'll find it is defined in the default logging config using standard Python logging: 'django': { 'handlers': ['console', 'mail_admins'], 'level': 'INFO', } As you can guess, the mail_admins handler is the one that does the work. It is defined as: … -
Disabling Error Emails in Django
One of Django's nice "batteries included" features is the ability to send emails when an error is encountered. This is a great feature for small sites where minor problems would otherwise go unnoticed. Once your site start getting lots of traffic, however, the feature turns into a liability. An error might fire off thousands of emails in rapid succession. Not only will this put extra load on your web servers, but you could also take down (or get banned from) your email server in the process. One of the first things you want to do when setting up a high-traffic Django site is replace the default error email functionality with an error reporting service like Sentry. Once you've got Sentry setup, what's the best way to disable error emails? Unfortunately, there isn't one simple answer. Let's dig into why... How does Django Send the Emails? First let's look at how Django sends those emails. If you look at the source, you'll find it is defined in the default logging config using standard Python logging: 'django': { 'handlers': ['console', 'mail_admins'], 'level': 'INFO', } As you can guess, the mail_admins handler is the one that does the work. It is defined as: … -
Custom Analytics with Django
Analytics can make our service... -
The Almighty Pause Container
When checking out the nodes of your Kubernetes cluster, you may have noticed some containers called "pause" running when you do a `docker ps` on the node. $ docker ps CONTAINER ID IMAGE COMMAND ... ... 3b45e983c859 gcr.io/google_containers/pause-amd64:3.0 "/pause" ... ... dbfc35b00062 gcr.io/google_containers/pause-amd64:3.0 "/pause" ... ... c4e998ec4d5d gcr.io/google_containers/pause-amd64:3.0 "/pause" ... ... 508102acf1e7 gcr.io/google_containers/pause-amd64:3.0 "/pause[...]