Category Archives: Uncategorized

Python PIP, Can’t connect to HTTPS URL because the SSL module is not available.

If you’re experiencing this error when ‘pip install’-ing or starting up a Docker container, then this post may be of use.

pip install -r requirements-dev.txt
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jb-bayes/
docker-compose -f docker-compose.yml up -d mysql redis
Traceback (most recent call last):
  File "/home/work/.pyenv/versions/bayes-service/bin/docker-compose", line 5, in <module>
  ...
  ...
    from .. import tls
  File "/home/work/.pyenv/versions/3.9.2/envs/bayes-service/lib/python3.9/site-packages/docker/tls.py", line 2, in <module>
    import ssl
  File "/home/work/.pyenv/versions/3.9.2/lib/python3.9/ssl.py", line 98, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: libssl.so.1.1: cannot open shared object file: No such file or directory

This is likely due to libssl being installed, then your python virtualenv env created, then something happening to libssl, such as being upgraded/removed.

The first step to resolve is to ensure you have libssl installed.

Download it from here

Then install:

wget http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0-dev_1.0.2n-1ubuntu5.9_amd64.deb

wget http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.7_amd64.deb

sudo apt install ./libssl1.0.0_1.0.2n-1ubuntu5.9_amd64.deb 
sudo apt install ./libssl1.0-dev_1.0.2n-1ubuntu5.9_amd64.deb

Once this is done you’ll need to reinstall the Python version you were using when the error originally occurred. In my case I’m using pyenv so had to take the below steps:

pyenv uninstall 3.9.2
pyenv install 3.9.2
pip install -r requirements-dev.txt
# Now installs without issue

If you’re not using pyenv, then however you installed Python remove it and reinstall.

How to connect to Cisco Meraki VPN on Ubuntu/Pop OS

I spent a long time getting this working myself so thought it best to save the steps somewhere. Be sure to follow every step exactly.

1) Run the following terminal commands.

sudo add-apt-repository ppa:nm-l2tp/network-manager-l2tp
sudo apt-get update 
sudo apt-get install network-manager-l2tp
sudo apt-get install network-manager-l2tp-gnome

2) Now Reboot your computer.

3) Open Network settings, click “+” to add a VPN connection of type “Layer 2 Tunneling Protocol (L2TP)”.

4) Enter a name for it.

5) Enter the hostname into the “Gateway” field.

6) Enter your username and password (click the <?> icon next to it first and select an option)

7) Click “IPSec Settings” and enter them as in the screenshot below.

8) Save/close the settings window(s).

9) Enter the following terminal commands:

sudo service xl2tpd stop
sudo systemctl disable xl2tpd

10) Re-open the Network settings and attempt to connect to your newly created VPN, it should now work. If it doesn’t check out tail -f /var/log/*.log for information on what could be wrong.

Compounding time-saving command-line tricks for Software Developers

First, find out what your most commonly used commands are, by running the below command.

history |  awk '{print $1}' | sort | uniq -c  | sort -nr | head -n 15

For me, it was the following.

    431 git
    122 ptw
    109 yarn
    100 sudo
     65 cd
     51 docker
     36 find
     34 rm
     31 pip
     31 ls
     23 cat
     20 pyenv
     20 kill
     16 gpg

To save time, over time. We can either try to remove commands or shorten the amount we need to type to execute them.

In my case, I added shell aliases for the following commands to shorten them.

g -> git
d -> docker
y -> yarn
f -> find

Sudo has to stay, but we can remove the need to have to keep typing in a password by taking the following steps.

  1. Type ‘sudo visudo
  2. In the file which opens in your editor, add the following line to the bottom, replacing ‘<your_username>’ with your system login username.

<your_username> ALL=(ALL) NOPASSWD: ALL

After this, you can type ‘sudo whatever command’ in a terminal without being prompted for the password.


That’s it. Although these changes seem minor, over time, you can save yourself a significant amount of time using these methods.

Do you have any tips to speed up your workflow and save time? If so, share them below!

One Python Development Setup to Rule Them All

Well, not quite. But here’s what I’ve settled on after working with Python constantly since 2012.

Virtualenv and Python version management

pyenv. I consider this the best as I can install and switch between any Python version (including conda) I like with ease. If you need this functionality and you’re not using pyenv, you’re missing out.

Installing Packages & managing their versions

I use pip. New tools have waded-in in recent years, but pip works for me and is still the most widely used tool.

Along with pip, I use requirements.txt (with fixed versions if it’s an application repo, supported version ranges if it’s a package/shared library project). I use requirements-dev.txt to hold development/test packages.

Releasing Packages

I use zest.releaser. This takes the pain out of releasing packages, automating the version number bumping, HISTORY updates, tagging etc.

IDE Setup

I use VSCode, because it’s fast and easy to use from day one.
I use an array of extensions. One, in particular, makes this choice a no brainer.

  • Settings Sync” this keeps your editor config in the ‘cloud’ meaning you can log in to any computer and sync that VSCode installation and you’re good to go in seconds.

I use the built-in terminal emulator within VSCode, along with Fish shell. I use Fish for its intelligent tab completion.

Testing

I use pytest for all Python unit and integration tests. It’s the industry standard right now and much better than the alternatives (nose, unittest).

To save a huge amount of time while doing TDD I use this tool avoid switching around my development environment.


That’s it, it’s quite boring but has stood the test of time. I hope this is useful to someone, if you have any suggested improvements please comment below, or Tweet me!

Supercharge your Python testing workflow

When writing unit tests in Python you may find yourself switching back and forth between your code editor window and terminal to re-run your tests.

You can avoid this by using inotify. In short, it can re-run your tests when you change any Python files in your project. Here are some little scripts to help you do that.

First, install:

sudo apt install -y inotify-tools

If you’re using bash, add this to your ~/.bashrc and restart your terminal:

function ptw() {
    pytest $1
    while find -name "*.py" | inotifywait -e close_write --fromfile - ;
        do pytest $1
    done
}

If you’re using Fish, add to ~/.config/fish/config.fish and restart your terminal:

function ptw
    pytest $argv
    while find -name "*.py" | inotifywait -e close_write --fromfile - ;
        pytest $argv
    end
end

Now instead of running your tests like:

pytest test_stuff.py

Just do:

ptw test_stuff.py

When the tests finish you’ll see the process does not exit, and when you edit a .py file in your project the tests will automatically re-run.

This saves a tonne of time, enjoy.


Fix: apt-get update hanging within Ubuntu Docker container

I experienced this issue in a Docker container, hanging at this point:

Step 5/24 : RUN apt-get update && apt-get install -y --no-install-recommends         cuda-cudart-$CUDA_PKG_VERSION         cuda-compat-10-1 &&     ln -s cuda-10.1 /usr/local/cuda &&     rm -rf /var/lib/apt/lists/*
 ---> Running in 3cb10279744e
Ign:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:4 http://deb.debian.org/debian stretch Release [118 kB]
Get:5 http://deb.debian.org/debian stretch Release.gpg [2434 B]
Get:6 http://deb.debian.org/debian stretch-updates/main amd64 Packages [11.1 kB]
Get:7 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [485 kB]
Get:8 http://deb.debian.org/debian stretch/main amd64 Packages [7084 kB]

Strace showed it waiting on data from a socket:

$ ps -faux|less|grep docker-compose
testuser  8098  0.4  0.4 115644 39016 pts/0    S+   08:56   0:00  |                   \_ /usr/bin/python3 /usr/local/bin/docker-compose build app
testuser  8402  0.0  0.0  13136  1084 pts/1    S+   08:59   0:00              \_ grep docker-compose
$ sudo strace -fp 8098
strace: Process 8098 attached
recvfrom(6, 

I narrowed down the issue to a https apt sources.list entry I had added and the fix was to install the following packages before that sources update:

apt-get install -y --no-install-recommends gnupg2 curl ca-certificates apt-transport-https

Everything guide for the privacy conscious

In 2019 most people’s data is held across cloud providers, free to be sold or subpoenaed at any time.

Here are my personal choices to limit this, be it in vain or not it helps me sleep better at night.

Email

Use Proton Mail, emails are decrypted in the browser using your key (password). iOS & Android apps.

Downsides: Limited features, apps only allow single user login at once.

File syncing

Use Tresorit, end-to-end encrypted. Unlike similar services this one has a decent mobile app for iOS and Android devices so you can sync & backup your Photos automatically.

Calendar syncing

Use a self-hosted Next Cloud running with the Calendar app enabled. This will give you a CalDAV server which most Calendar applications can sync with.

Note taking

Use Standard Notes, end-to-end encrypted syncing with apps for Mac, Windows, Linux, iOS, Android.

VPN

If you don’t want to host your own VPN then use iPredator. Founded by Peter Sunde this is the only provider I would consider trusting.

If you’d like to host your own, use Algo.

Password management

Use LastPass, end-to-end encrypted with plugins for Firefox, Chrome & Android.

Instant messaging

Use Signal, end-to-end encrypted by default, unlike other services.


Do you have any better suggestions for the above? Comment below!

2017 Macbook Pro vs Lenovo X1 Carbon

I’ve been using a latest model Macbook Pro for the last 6 months or so now and have recently got my hands on a Lenovo X1 Carbon (6th gen). So here’s a comparison of the two from a software engineering perspective.

Macbook Pro specification:

  • 3.5GHz Intel i7 CPU
  • 16GB RAM
  • 1TB SSD
  • 13in screen (retina)
  • Priced around £2800

X1 Carbon specification:

  • Intel Core i7-8650U Processor (8MB Cache, up to 4.2GHz)
  • 16GB RAM
  • 1TB SSD
  • 14in screen (HDR)
  • Priced around £2600

The main worry I had with the X1 Carbon was that the screen wouldn’t have the vibrant brightness of the Mac. However I was pleased to find the X1 screen brightness matches that of the Mac.

The Mac keyboard was a true pain point. Every couple of days a random key would just stop functioning. This is clearly a design flaw, with small amounts of dust under the keys causing them to not work. Apple posted this “fix” for the issue which works but is a chore to keep blasting the keyboard to keep it working,

The X1 Carbon keyboard on the other hand has been a dream to work with. The keys are much more raised up than those of the Mac which feels nicer to type on. It’s also better to have a REAL “Esc” key to reach for, which is missing from the Mac as it’s now built into the touch-bar.

Ports. The Mac has a headphone jack along with a bunch of USB-C ports which forced me to spend on additional adapters. The X1 Carbon comes with USB, USB-C, HDMI, headphone jack and an SD card slot on the back. Enough said.

Cooling. The Mac external design looks great, but it lacks some good ventilation. Under high load the base of the Aluminum case can get quite hot — not good when you’re not using the laptop on a desk. The X1 Carbon has some big beefy vents on the side which seem to do a good job — just be careful not to place your fingers over the vents as they chuck out the hot air.

Overall. Both are great laptops, but due to the ports issue (which Apple are taking the piss with) along with the broken keyboard design means I’m going to make the X1 Carbon my primary laptop going forward.

Hopefully this was useful to someone. Post a comment below if you have any experience with either laptop.

Find which process is utilizing a port and kill it

Step 1. Use the list open files command to find the process using port 5434, for example.

sudo lsof -i tcp:5434

E.g. output:

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
com.docke 55746 work   18u  IPv4 0x97cf67a7c0ce1ffb      0t0  TCP *:sgi-arrayd (LISTEN)
com.docke 55746 work   19u  IPv6 0x97cf67a7dd4c56d3      0t0  TCP localhost:sgi-arrayd (LISTEN)

Step 2. Kill the process by PID:

sudo kill -12 55746

Done!

How to debug a “Segmentation fault” in Python

Sometimes you’ll get a segmentation fault in Python and your process will crash, this is due to a C module attempting to access memory beyond reach.

Output in this case will be very limited:

Segmentation fault

To get a full traceback we need to enable the Python faulthandler module.

First add the following to the top of your module.

import faulthandler; faulthandler.enable()

Then re-run your program with the faulthandler startup flag.

Passed as an argument.

# pass as an argument
python -Xfaulthandler my_program.py

# Or as an environment variable.
PYTHONFAULTHANDLER=1 python my_program.py

Now you will get a detailed traceback.

Fatal Python error: Segmentation fault
Current thread 0x00007f0a49caa700 (most recent call first):
File "", line 219 in _call_with_frames_removed
File "", line 922 in create_module
File "", line 571 in module_from_spec
File "", line 658 in _load_unlocked
File "", line 684 in _load
File "/usr/local/lib/python3.6/imp.py", line 343 in load_dynamic
File "/usr/local/lib/python3.6/imp.py", line 243 in load_module
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24 in swig_import_helper
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/__init__.py", line 24 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 5 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/__init__.py", line 83 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/conv_utils.py", line 9 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/__init__.py", line 6 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/__init__.py", line 3 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 941 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/forecasting.py", line 7 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/customer_operations.py", line 12 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/app.py", line 13 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 994 in _gcd_import
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126 in import_module
File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101 in import_from_cwd
File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 506 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 355 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 503 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 481 in setup_app_from_commandline
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279 in execute_from_commandline
...
Segmentation fault

Happy debugging!