Tag Archives: python

One Python Development Setup to Rule Them All

Well, not quite. But here’s what I’ve settled on after working with Python constantly since 2012.

Virtualenv and Python version management

pyenv. I consider this the best as I can install and switch between any Python version (including conda) I like with ease. If you need this functionality and you’re not using pyenv, you’re missing out.

Installing Packages & managing their versions

I use pip. New tools have waded-in in recent years, but pip works for me and is still the most widely used tool.

Along with pip, I use requirements.txt (with fixed versions if it’s an application repo, supported version ranges if it’s a package/shared library project). I use requirements-dev.txt to hold development/test packages.

Releasing Packages

I use zest.releaser. This takes the pain out of releasing packages, automating the version number bumping, HISTORY updates, tagging etc.

IDE Setup

I use VSCode, because it’s fast and easy to use from day one.
I use an array of extensions. One, in particular, makes this choice a no brainer.

  • Settings Sync” this keeps your editor config in the ‘cloud’ meaning you can log in to any computer and sync that VSCode installation and you’re good to go in seconds.

I use the built-in terminal emulator within VSCode, along with Fish shell. I use Fish for its intelligent tab completion.

Testing

I use pytest for all Python unit and integration tests. It’s the industry standard right now and much better than the alternatives (nose, unittest).

To save a huge amount of time while doing TDD I use this tool avoid switching around my development environment.


That’s it, it’s quite boring but has stood the test of time. I hope this is useful to someone, if you have any suggested improvements please comment below, or Tweet me!

How to migrate from multi-version Python Travis-CI builds to Gitlab CI

With Travis-CI you can setup a CI build to run against multiple Python versions fairly easily.

.travis.yml

sudo: false
language: python
python:
    - 2.7
    - 3.6
env:
  - TOXENV=py-normal
install: pip install tox
script: tox

tox.ini

[tox]
envlist = py{27,36}-normal

[testenv]
commands =
    pytest

deps =
    -rtest_requirements.txt

You can achieve something similar with Gitlab CI through the following .gitlab-ci.yml configuration. Your tox.ini can remain the same.

before_script:
  # Install pyenv
  - apt-get update
  - apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev
  - git clone https://github.com/pyenv/pyenv.git ~/.pyenv
  - export PYENV_ROOT="$HOME/.pyenv"
  - export PATH="$PYENV_ROOT/bin:$PATH"
  - eval "$(pyenv init -)"
  # Install tox
  - pip install tox

test:python27:
  script:
  - pyenv install 2.7.14
  - pyenv shell 2.7.14
  - tox -e py27-normal

test:python36:
  script:
  - pyenv install 3.6.4
  - pyenv shell 3.6.4
  - tox -e py36-normal

The only downside with this is the extra time it takes to install pyenv and the interpreter of choice. A small price to pay to free your project from Github ;)

How to debug a “Segmentation fault” in Python

Sometimes you’ll get a segmentation fault in Python and your process will crash, this is due to a C module attempting to access memory beyond reach.

Output in this case will be very limited:

Segmentation fault

To get a full traceback we need to enable the Python faulthandler module.

First add the following to the top of your module.

import faulthandler; faulthandler.enable()

Then re-run your program with the faulthandler startup flag.

Passed as an argument.

# pass as an argument
python -Xfaulthandler my_program.py

# Or as an environment variable.
PYTHONFAULTHANDLER=1 python my_program.py

Now you will get a detailed traceback.

Fatal Python error: Segmentation fault
Current thread 0x00007f0a49caa700 (most recent call first):
File "", line 219 in _call_with_frames_removed
File "", line 922 in create_module
File "", line 571 in module_from_spec
File "", line 658 in _load_unlocked
File "", line 684 in _load
File "/usr/local/lib/python3.6/imp.py", line 343 in load_dynamic
File "/usr/local/lib/python3.6/imp.py", line 243 in load_module
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24 in swig_import_helper
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/__init__.py", line 24 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 5 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/__init__.py", line 83 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/conv_utils.py", line 9 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/__init__.py", line 6 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/__init__.py", line 3 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 941 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/forecasting.py", line 7 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/customer_operations.py", line 12 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/app.py", line 13 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 994 in _gcd_import
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126 in import_module
File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101 in import_from_cwd
File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 506 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 355 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 503 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 481 in setup_app_from_commandline
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279 in execute_from_commandline
...
Segmentation fault

Happy debugging!

Read a file in chunks in Python

This article is just to demonstrate how to read a file in chunks rather than all at once.

This is useful for a number of cases, such as chunked uploading or encryption purposes, or perhaps where the file you want to interact with is larger than your machine memory capacity.

# chunked file reading
from __future__ import division
import os

def get_chunks(file_size):
    chunk_start = 0
    chunk_size = 0x20000  # 131072 bytes, default max ssl buffer size
    while chunk_start + chunk_size < file_size:
        yield(chunk_start, chunk_size)
        chunk_start += chunk_size

    final_chunk_size = file_size - chunk_start
    yield(chunk_start, final_chunk_size)

def read_file_chunked(file_path):
    with open(file_path) as file_:
        file_size = os.path.getsize(file_path)

        print('File size: {}'.format(file_size))

        progress = 0

        for chunk_start, chunk_size in get_chunks(file_size):

            file_chunk = file_.read(chunk_size)

            # do something with the chunk, encrypt it, write to another file...

            progress += len(file_chunk)
            print('{0} of {1} bytes read ({2}%)'.format(
                progress, file_size, int(progress / file_size * 100))
            )

if __name__ == '__main__':
    read_file_chunked('some-file.gif')

Also available as a Gist (https://gist.github.com/richardasaurus/21d4b970a202d2fffa9c)

The above will output:

File size: 698837
131072 of 698837 bytes read (18%)
262144 of 698837 bytes read (37%)
393216 of 698837 bytes read (56%)
524288 of 698837 bytes read (75%)
655360 of 698837 bytes read (93%)
698837 of 698837 bytes read (100%)

Hopefully handy to someone. This of course isn't the only way, you could also use `file.seek` in the standard library to target chunks.

Concurrent Jenkins builds of a Django application

If you try to run multiple Jenkins builds of a single Django project on Jenkins out of the box you might be met with a message similar to:

Got an error creating the test database: database "test_projectdb" already exists

Got an error recreating the test database: database "test_projectdb" is being accessed by other users
DETAIL:  There is 1 other session using the database.

To fix this you need to edit the ‘DATABASES’ Dictionary within your Django project settings module, adding another key ‘TEST_NAME’.

TEST_NAME is the name of the test database Django will create when running your tests with manage.py.

We can make this name unique by adding the following function to our Django setting module:

def get_test_db_name():
    md5 = hashlib.md5()
    md5.update(os.environ.get('BUILD_TAG', b'no-tag'))
    return md5.hexdigest()

(This will take the unique BUILD_TAG environment variable set by Jenkins and md5 it)

And then calling it within the DATABASES Dictionary:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'projectdb',
        'USER': 'django',
        'PASSWORD': 'django',
        'HOST': '127.0.0.1',
        'PORT': '',
        'TEST_NAME': get_test_db_name(),
    }
}

That’s it, Jenkins should now work fine with concurrent builds of your application.

 

Separation of logic in Django Projects

Currently I work mostly on a large Django code-base 4+ years old in which business logic is tangled throughout the three MVC components.

Given such a large framework, developers often forget how to write well-organized Python business logic code they’re definitely capable of given the absence of the framework.

As someone who has worked on projects with these symptoms, as well as other older code bases before my Django days, the situation isn’t as bad as some very old PHP projects I’ve seen. Anyway, here are some tips which can be applied to Django applications to aid in organization overall.

Keep only database code within the models module

You’ll often see lengthy ORM queries randomly plastered throughout an application. Keeping these within the model class makes everything more maintainable.

  • Place ORM queries within your models module
  • Wrap up the queries you need using these features
  • Keep business logic out of here

Create modules for business logic

As you would with a regular Python program. Create your own modules outside of Django component structure.
Make your logic functions responsible only for logic, as in not caring about the presentation or data layer (use dependency injection).

Views should be light

Your views should only be used to glue things together, linking requests to forms, forms to your business logic and outputs to templates.
The view then becomes a simple description of how a feature is coupled.

Override forms for validation

  • Override the Django form methods to add any complex custom validation.

Keeping all of your validation code inside the forms module means errors can always to tied back to the individual inputs.


A result of the above rules is increased testability, easier adaptability and of course it’s in-keeping with the separation of concerns design principal.

Navigation active state in Django

Here’s a clean way to display a navigation menu item’s active state in Django.

Wherever the app you’re doing this for is located you’ll have an urls.py. Ensure you have the name set within each url group.

from django.conf.urls import patterns, url
from apps.pages import views

urlpatterns = patterns('',
    url(r'^pages/$', views.pages.index, name='pages.index'),
    url(r'^pages/about$', views.pages.about, name='pages.about'),
)

Next create a directory called templatetags within your app folder.
Add to it a blank __init__.py and nav_active.py, giving it the below content.

from django.core.urlresolvers import resolve
from django.template import Library

register = Library()

@register.simple_tag
def nav_active(request, url):
    """
    In template: {% nav_active request "url_name_here" %}
    """
    url_name = resolve(request.path).url_name
    if url_name == url:
        return "active"
    return ""

# nav_active() will check the web request url_name and compare it 
# to the named url group within urls.py, 
# setting the active class if they match.

Now to finish up, in your template .html file you need to load in the template tag and add it to each navigation item.

{% load nav_active %}


How to use Enums for Django Field.choices

In Django when using the choices parameter on a form field the format passed in must be as follows:

# within your models.Model class...
STUDENT_TYPE_CHOICES = (
    ('0', 'freshman'),
    ('1', 'sophomore'),
    ('2', 'junior'),
    ('3', 'senior'),
)
student_type = models.CharField(max_length=1, choices=STUDENT_TYPE_CHOICES)

This means elsewhere in your code if you want to specify a choice field value, you’d have to enter the first slot of the tuple’s value, e.g.:

junior_students = Student.objects.filter(student_type='2')

This is pretty terrible since it’s hardcoded in our source, possibly over many files.

How to fix this mess:

In my project I added common/utils.py containing the following:

from enum import Enum

class ChoiceEnum(Enum):
    @classmethod
    def choices(cls):
        return tuple((i.name, i.value) for i in cls)

That’s the hard work over.

Now when you create your field with choices:

from common.utils import ChoiceEnum

class StudentTypes(ChoiceEnum):
    freshman = 0
    sophomore = 1
    junior = 2
    senior = 3

# within your models.Model class...
class Student(models.Model):
    student_type = models.CharField(max_length=1, choices=StudentTypes.choices())

 

Now if we need to access StudentTypes from elsewhere in our source code, we can simply:

junior_students = Student.objects.filter(student_type=StudentTypes.junior.value)

 

That’s it. If anyone knows of a nicer way feel free to comment below.

Setting up NGINX + Django + uWSGI (a tutorial that actually works)

So after reading the various tutorials online for setting up NGINX + Django + uWSGI and all of them not working correctly, I decided to write my own.

This tutorial was tested on a blank install of Ubuntu Server 12.04 LTS 64-bit, if you follow the steps carefully in the correct order all should be well :)

1. Add a new user, give them sudo privileges and switch to that user, below I’ve named mine “user”.

sudo adduser user
sudo adduser user sudo
su user

 

2. Ensure your system hostname is set to localhost

sudo echo "localhost" > /etc/hostname
sudo hostname localhost

 

3. Since this is a new install, update the system.

sudo apt-get update
sudo apt-get upgrade

 

4. Install python, virtual environment builder and python dev

sudo apt-get install python
sudo apt-get install python-virtualenv
sudo apt-get install python2.7-dev

 

5. Install and start the NGINX web server

sudo apt-get install nginx
sudo service nginx start

 

6. Install uWSGI

sudo apt-get install uwsgi

 

7. Setup a Django project

sudo mkdir /var/www
sudo mkdir /var/www/example.com
cd /var/www/example.com
sudo mkdir venv conf src logs

 

This will give the below pictured folder structure
folder structure

 

8. Set-up the virtual environment and activate it

sudo virtualenv /var/www/example.com/venv
source /var/www/example.com/venv/bin/activate

 

9. Install Django

sudo pip install django

 

10. Change to the “src” directory, then copy your Django project files into it

cd /var/www/example.com/src

 

10. Create your uwsgi.ini config file, with the below content

sudo nano /var/www/example.com/conf/uwsgi.ini
[uwsgi]
# variables
projectname = example_project
projectdomain = example.com
base = /var/www/example.com

# config
plugins = python
master = true
protocol = uwsgi
env = DJANGO_SETTINGS_MODULE=%(projectname).settings
pythonpath = %(base)/src/%(projectname)
module = %(projectname).wsgi
socket = 127.0.0.1:8889
logto = %(base)/logs/uwsgi.log
#below line runs it as a daemon in background
daemonize = /var/log/uwsgi/example_project.log

 

11. Create an NGINX config file for this domain, with the below content

sudo nano /var/www/example.com/conf/nginx.conf
server {
    listen 80;
    server_name example.com www.example.com;
    root /var/www/example.com/src/example_project;
    access_log /var/www/example.com/logs/access.log;
    error_log /var/www/example.com/logs/error.log;

    location /static/ { # STATIC_URL
        alias /var/www/example.com/src/static/; # STATIC_ROOT
        expires 30d;
    }

    location /media/ { # MEDIA_URL
        alias /var/www/example.com/src/media/; # MEDIA_ROOT
        expires 30d;
    }

    location / {
        include uwsgi_params;
        uwsgi_pass 127.0.0.1:8889;
    }
}

 

12. Edit the main nginx.conf to import our domain conf file, see below content as a guide

sudo nano /etc/nginx/nginx.conf
user    www-data;
# ...
http {
    # ...
    include /var/www/*/conf/nginx.conf;
    # ...
}

 

13. Restart NGINX (to load apply our config changes)

sudo service nginx restart

 

14. Install MySQL and secure it

sudo apt-get install mysql-server
sudo mysql_secure_installation

 

15. Install Python MySQL and uWSGI plugins

sudo apt-get install python-mysqldb
sudo apt-get install uwsgi-plugin-python

 

16. Install south (optional)

sudo pip install south

 

17. Test that uWSGI is working

sudo uwsgi --ini /var/www/example.com/conf/uwsgi.ini

If you visit your site it should now show django. If it doesn’t common causes are:

  • ALLOWED_HOSTS in settings.py isn’t set
  • DEBUG isn’t off in settings.py
  • Database isn’t configured

 

18. Setup uWSGI to run on system boot

Create the following file, with the below content

sudo nano /etc/init/uwsgi.conf
# Emperor uWSGI script

description "uWSGI Emperor"
start on runlevel [2345]
stop on runlevel [06]

exec uwsgi --master --die-on-term --emperor /var/www/example.com/conf/uwsgi.ini

 

19. Now reboot the server and navigate to your website

sudo reboot