2017 Macbook Pro vs Lenovo X1 Carbon

I’ve been using a latest model Macbook Pro for the last 6 months or so now and have recently got my hands on a Lenovo X1 Carbon (6th gen). So here’s a comparison of the two from a software engineering perspective.

Macbook Pro specification:

  • 3.5GHz Intel i7 CPU
  • 16GB RAM
  • 1TB SSD
  • 13in screen (retina)
  • Priced around £2800

X1 Carbon specification:

  • Intel Core i7-8650U Processor (8MB Cache, up to 4.2GHz)
  • 16GB RAM
  • 1TB SSD
  • 14in screen (HDR)
  • Priced around £2600

The main worry I had with the X1 Carbon was that the screen wouldn’t have the vibrant brightness of the Mac. However I was pleased to find the X1 screen brightness matches that of the Mac.

The Mac keyboard was a true pain point. Every couple of days a random key would just stop functioning. This is clearly a design flaw, with small amounts of dust under the keys causing them to not work. Apple posted this “fix” for the issue which works but is a chore to keep blasting the keyboard to keep it working,

The X1 Carbon keyboard on the other hand has been a dream to work with. The keys are much more raised up than those of the Mac which feels nicer to type on. It’s also better to have a REAL “Esc” key to reach for, which is missing from the Mac as it’s now built into the touch-bar.

Ports. The Mac has a headphone jack along with a bunch of USB-C ports which forced me to spend on additional adapters. The X1 Carbon comes with USB, USB-C, HDMI, headphone jack and an SD card slot on the back. Enough said.

Cooling. The Mac external design looks great, but it lacks some good ventilation. Under high load the base of the Aluminum case can get quite hot — not good when you’re not using the laptop on a desk. The X1 Carbon has some big beefy vents on the side which seem to do a good job — just be careful not to place your fingers over the vents as they chuck out the hot air.

Overall. Both are great laptops, but due to the ports issue (which Apple are taking the piss with) along with the broken keyboard design means I’m going to make the X1 Carbon my primary laptop going forward.

Hopefully this was useful to someone. Post a comment below if you have any experience with either laptop.

How to migrate from multi-version Python Travis-CI builds to Gitlab CI

With Travis-CI you can setup a CI build to run against multiple Python versions fairly easily.

.travis.yml

sudo: false
language: python
python:
    - 2.7
    - 3.6
env:
  - TOXENV=py-normal
install: pip install tox
script: tox

tox.ini

[tox]
envlist = py{27,36}-normal

[testenv]
commands =
    pytest

deps =
    -rtest_requirements.txt

You can achieve something similar with Gitlab CI through the following .gitlab-ci.yml configuration. Your tox.ini can remain the same.

before_script:
  # Install pyenv
  - apt-get update
  - apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev
  - git clone https://github.com/pyenv/pyenv.git ~/.pyenv
  - export PYENV_ROOT="$HOME/.pyenv"
  - export PATH="$PYENV_ROOT/bin:$PATH"
  - eval "$(pyenv init -)"
  # Install tox
  - pip install tox

test:python27:
  script:
  - pyenv install 2.7.14
  - pyenv shell 2.7.14
  - tox -e py27-normal

test:python36:
  script:
  - pyenv install 3.6.4
  - pyenv shell 3.6.4
  - tox -e py36-normal

The only downside with this is the extra time it takes to install pyenv and the interpreter of choice. A small price to pay to free your project from Github ;)

Find which process is utilizing a port and kill it

Step 1. Use the list open files command to find the process using port 5434, for example.

sudo lsof -i tcp:5434

E.g. output:

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
com.docke 55746 work   18u  IPv4 0x97cf67a7c0ce1ffb      0t0  TCP *:sgi-arrayd (LISTEN)
com.docke 55746 work   19u  IPv6 0x97cf67a7dd4c56d3      0t0  TCP localhost:sgi-arrayd (LISTEN)

Step 2. Kill the process by PID:

sudo kill -12 55746

Done!

How to debug a “Segmentation fault” in Python

Sometimes you’ll get a segmentation fault in Python and your process will crash, this is due to a C module attempting to access memory beyond reach.

Output in this case will be very limited:

Segmentation fault

To get a full traceback we need to enable the Python faulthandler module.

First add the following to the top of your module.

import faulthandler; faulthandler.enable()

Then re-run your program with the faulthandler startup flag.

Passed as an argument.

# pass as an argument
python -Xfaulthandler my_program.py

# Or as an environment variable.
PYTHONFAULTHANDLER=1 python my_program.py

Now you will get a detailed traceback.

Fatal Python error: Segmentation fault
Current thread 0x00007f0a49caa700 (most recent call first):
File "", line 219 in _call_with_frames_removed
File "", line 922 in create_module
File "", line 571 in module_from_spec
File "", line 658 in _load_unlocked
File "", line 684 in _load
File "/usr/local/lib/python3.6/imp.py", line 343 in load_dynamic
File "/usr/local/lib/python3.6/imp.py", line 243 in load_module
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24 in swig_import_helper
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/__init__.py", line 24 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 5 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/__init__.py", line 83 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/conv_utils.py", line 9 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/__init__.py", line 6 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/__init__.py", line 3 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 941 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/forecasting.py", line 7 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/customer_operations.py", line 12 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/app.py", line 13 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 994 in _gcd_import
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126 in import_module
File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101 in import_from_cwd
File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 506 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 355 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 503 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 481 in setup_app_from_commandline
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279 in execute_from_commandline
...
Segmentation fault

Happy debugging!

Read a file in chunks in Python

This article is just to demonstrate how to read a file in chunks rather than all at once.

This is useful for a number of cases, such as chunked uploading or encryption purposes, or perhaps where the file you want to interact with is larger than your machine memory capacity.

# chunked file reading
from __future__ import division
import os

def get_chunks(file_size):
    chunk_start = 0
    chunk_size = 0x20000  # 131072 bytes, default max ssl buffer size
    while chunk_start + chunk_size < file_size:
        yield(chunk_start, chunk_size)
        chunk_start += chunk_size

    final_chunk_size = file_size - chunk_start
    yield(chunk_start, final_chunk_size)

def read_file_chunked(file_path):
    with open(file_path) as file_:
        file_size = os.path.getsize(file_path)

        print('File size: {}'.format(file_size))

        progress = 0

        for chunk_start, chunk_size in get_chunks(file_size):

            file_chunk = file_.read(chunk_size)

            # do something with the chunk, encrypt it, write to another file...

            progress += len(file_chunk)
            print('{0} of {1} bytes read ({2}%)'.format(
                progress, file_size, int(progress / file_size * 100))
            )

if __name__ == '__main__':
    read_file_chunked('some-file.gif')

Also available as a Gist (https://gist.github.com/richardasaurus/21d4b970a202d2fffa9c)

The above will output:

File size: 698837
131072 of 698837 bytes read (18%)
262144 of 698837 bytes read (37%)
393216 of 698837 bytes read (56%)
524288 of 698837 bytes read (75%)
655360 of 698837 bytes read (93%)
698837 of 698837 bytes read (100%)

Hopefully handy to someone. This of course isn’t the only way, you could also use `file.seek` in the standard library to target chunks.

Getting console.log errors with Selenium, PhantomJS in Python

So I had some functional tests passing on my workstation, but when pushed to CI environment they would fail with an “ElementNotVisibleException” exception, because scripts which created the element weren’t doing their job.

I wanted to view the browser console.log to get some clues to what went wrong on the front-end.

The selenium docs state to use:

driver.get_log('browser')

But in my case that returned an empty list, not very useful.

I’ve found if you use log type of “har”, not in the docs:

driver.get_log('har')

It will return a bunch information, including, if you look carefully, some “NOT FOUND” errors for for requests triggered by Javascript code.

[{
    'timestamp': 1436900661766,
    'message': '{"log":{"version":"1.2","creator":{"name":"PhantomJS","version":"2.0.0"},"pages":[{"startedDateTime":"2015-07-14T19:01:40.795Z","id":"http://myapp.domain.com:8081/people/1","title":"John Smith - MyAppName","pageTimings":{"onLoad":2900}}],"entries":[{"startedDateTime":"2015-07-14T19:03:22.559Z","time":148,"request":{"method":"GET","url":"http://myapp.domain.com:8081/people/1","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Accept","value":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},{"name":"Cache-Control","value":"max-age=0"},{"name":"User-Agent","value":"Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.0.0 Safari/538.1"}],"queryString":[],"headersSize":-1,"bodySize":-1},"response":{"status":200,"statusText":"OK","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Date","value":"Tue, 14 Jul 2015 19:03:22 GMT"},{"name":"Server","value":"WSGIServer/0.1 Python/2.7.6"},{"name":"Vary","value":"Cookie"},{"name":"X-Frame-Options","value":"SAMEORIGIN"},{"name":"Content-Type","value":"text/html; charset=utf-8"},{"name":"Set-Cookie","value":"csrftoken=wFzWPTm9aVkGLtPuOCcc1tIs6ve5KosW; expires=Tue, 12-Jul-2016 19:03:22 GMT; Max-Age=31449600; Path=/"}],"redirectURL":"","headersSize":-1,"bodySize":5776,"content":{"size":5776,"mimeType":"text/html; charset=utf-8"}},"cache":{},"timings":{"blocked":0,"dns":-1,"connect":-1,"send":0,"wait":140,"receive":8,"ssl":-1},"pageref":"http://myapp.domain.com:8081/people/1"},{"startedDateTime":"2015-07-14T19:03:22.705Z","time":7,"request":{"method":"GET","url":"http://myapp.domain.com:8081/static/js/lib/bower_components/requirejs/require.js","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Accept","value":"*/*"},{"name":"Referer","value":"http://myapp.domain.com:8081/people/1"},{"name":"User-Agent","value":"Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.0.0 Safari/538.1"}],"queryString":[],"headersSize":-1,"bodySize":-1},"response":{"status":null,"statusText":"Error downloading http://myapp.domain.com:8081/static/js/lib/bower_components/requirejs/require.js - server replied: NOT FOUND","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Date","value":"Tue, 14 Jul 2015 19:03:22 GMT"},{"name":"Server","value":"WSGIServer/0.1 Python/2.7.6"},{"name":"X-Frame-Options","value":"SAMEORIGIN"},{"name":"Content-Type","value":"text/html"}],"redirectURL":"","headersSize":-1,"bodySize":125,"content":{"size":125,"mimeType":"text/html"}},"cache":{},"timings":{"blocked":0,"dns":-1,"connect":-1,"send":0,"wait":6,"receive":1,"ssl":-1},"pageref":"http://myapp.domain.com:8081/people/1"},{"startedDateTime":"2015-07-14T19:03:22.706Z","time":15,"request":{"method":"GET","url":"http://myapp.domain.com:8081/static/js/lib/bower_components/jquery-ui/themes/ui-lightness/jquery.ui.theme.css","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Accept","value":"text/css,*/*;q=0.1"},{"name":"Referer","value":"http://myapp.domain.com:8081/people/1"},{"name":"User-Agent","value":"Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.0.0 Safari/538.1"}],"queryString":[],"headersSize":-1,"bodySize":-1},"response":{"status":null,"statusText":"Error downloading http://myapp.domain.com:8081/static/js/lib/bower_components/jquery-ui/themes/ui-lightness/jquery.ui.theme.css - server replied: NOT FOUND","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Date","value":"Tue, 14 Jul 2015 19:03:22 GMT"},{"name":"Server","value":"WSGIServer/0.1 Python/2.7.6"},{"name":"X-Frame-Options","value":"SAMEORIGIN"},{"name":"Content-Type","value":"text/html"}],"redirectURL":"","headersSize":-1,"bodySize":154,"content":{"size":154,"mimeType":"text/html"}},"cache":{},"timings":{"blocked":0,"dns":-1,"connect":-1,"send":0,"wait":15,"receive":0,"ssl":-1},"pageref":"http://myapp.domain.com:8081/people/1"}]}}',
    'level': 'INFO'
}]

So with that I found JS assets weren’t being compiled in my CI environment and was able to go ahead and fix it :)

Hopefully that’s useful to someone out there.

Why is Programming Fun?

An extract from Fred Brooks’ (Frederick P. Brooks, Jr.) book, The Mythical Man-Month.

Why is programming fun? What delights may its practioner expect as his reward?

First is the sheer joy of making things. As the child delights in his mud pie, so the adult enjoys building things, especially things of his own design. I think this delight must be an image of God’s delight in making things, a delight shown in the distinctness and newness of each leaf and each snowflake.

Second is the pleasure of making things that are useful to other people. Deep within, we want others to use our work and to find it helpful. In this respect the programming system is not essentially different from the child’s first clay pencil holder “for Daddy’s office.”

Third is the fascination of fashioning complex puzzle-like objects of interlocking moving parts and watching them work in subtle cycles, playing out the consequences of principles built in from the beginning. The programmed computer has all the fascination of the pinball machine or the jukebox mechanism, carried to the ultimate.

Fourth is the joy of always learning, which springs from the nonrepeating nature of the task. In one way or another the problem is ever new, and its solver learns something: sometimes practical, sometimes theoretical, and sometimes both.

Finally, there is the delight of working in such a tractable medium. The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures. (…)
Yet the program construct, unlike the poet’s words, is real in the sense that it moves and works, producing visible outputs separately from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.

Programming then is fun because it gratifies creative longings built deep within us and delights sensibilities we have in common with all men.

Concurrent Jenkins builds of a Django application

If you try to run multiple Jenkins builds of a single Django project on Jenkins out of the box you might be met with a message similar to:

Got an error creating the test database: database "test_projectdb" already exists

Got an error recreating the test database: database "test_projectdb" is being accessed by other users
DETAIL:  There is 1 other session using the database.

To fix this you need to edit the ‘DATABASES’ Dictionary within your Django project settings module, adding another key ‘TEST_NAME’.

TEST_NAME is the name of the test database Django will create when running your tests with manage.py.

We can make this name unique by adding the following function to our Django setting module:

def get_test_db_name():
    md5 = hashlib.md5()
    md5.update(os.environ.get('BUILD_TAG', b'no-tag'))
    return md5.hexdigest()

(This will take the unique BUILD_TAG environment variable set by Jenkins and md5 it)

And then calling it within the DATABASES Dictionary:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'projectdb',
        'USER': 'django',
        'PASSWORD': 'django',
        'HOST': '127.0.0.1',
        'PORT': '',
        'TEST_NAME': get_test_db_name(),
    }
}

That’s it, Jenkins should now work fine with concurrent builds of your application.

 

Separation of logic in Django Projects

Currently I work mostly on a large Django code-base 4+ years old in which business logic is tangled throughout the three MVC components.

Given such a large framework, developers often forget how to write well-organized Python business logic code they’re definitely capable of given the absence of the framework.

As someone who has worked on projects with these symptoms, as well as other older code bases before my Django days, the situation isn’t as bad as some very old PHP projects I’ve seen. Anyway, here are some tips which can be applied to Django applications to aid in organization overall.

Keep only database code within the models module

You’ll often see lengthy ORM queries randomly plastered throughout an application. Keeping these within the model class makes everything more maintainable.

  • Place ORM queries within your models module
  • Wrap up the queries you need using these features
  • Keep business logic out of here

Create modules for business logic

As you would with a regular Python program. Create your own modules outside of Django component structure.
Make your logic functions responsible only for logic, as in not caring about the presentation or data layer (use dependency injection).

Views should be light

Your views should only be used to glue things together, linking requests to forms, forms to your business logic and outputs to templates.
The view then becomes a simple description of how a feature is coupled.

Override forms for validation

  • Override the Django form methods to add any complex custom validation.

Keeping all of your validation code inside the forms module means errors can always to tied back to the individual inputs.


A result of the above rules is increased testability, easier adaptability and of course it’s in-keeping with the separation of concerns design principal.