Category Archives: Uncategorized

How to debug a “Segmentation fault” in Python

Sometimes you’ll get a segmentation fault in Python and your process will crash, this is due to a C module attempting to access memory beyond reach.

Output in this case will be very limited:

Segmentation fault

To get a full traceback we need to enable the Python faulthandler module.

First add the following to the top of your module.

import faulthandler; faulthandler.enable()

Then re-run your program with the faulthandler startup flag.

Passed as an argument.

# pass as an argument
python -Xfaulthandler my_program.py

# Or as an environment variable.
PYTHONFAULTHANDLER=1 python my_program.py

Now you will get a detailed traceback.

Fatal Python error: Segmentation fault
Current thread 0x00007f0a49caa700 (most recent call first):
File "", line 219 in _call_with_frames_removed
File "", line 922 in create_module
File "", line 571 in module_from_spec
File "", line 658 in _load_unlocked
File "", line 684 in _load
File "/usr/local/lib/python3.6/imp.py", line 343 in load_dynamic
File "/usr/local/lib/python3.6/imp.py", line 243 in load_module
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24 in swig_import_helper
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/__init__.py", line 24 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 5 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/__init__.py", line 83 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/conv_utils.py", line 9 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/__init__.py", line 6 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/__init__.py", line 3 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 941 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/forecasting.py", line 7 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/customer_operations.py", line 12 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/app.py", line 13 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 994 in _gcd_import
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126 in import_module
File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101 in import_from_cwd
File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 506 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 355 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 503 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 481 in setup_app_from_commandline
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279 in execute_from_commandline
...
Segmentation fault

Happy debugging!

Getting console.log errors with Selenium, PhantomJS in Python

So I had some functional tests passing on my workstation, but when pushed to CI environment they would fail with an “ElementNotVisibleException” exception, because scripts which created the element weren’t doing their job.

I wanted to view the browser console.log to get some clues to what went wrong on the front-end.

The selenium docs state to use:

driver.get_log('browser')

But in my case that returned an empty list, not very useful.

I’ve found if you use log type of “har”, not in the docs:

driver.get_log('har')

It will return a bunch information, including, if you look carefully, some “NOT FOUND” errors for for requests triggered by Javascript code.

[{
    'timestamp': 1436900661766,
    'message': '{"log":{"version":"1.2","creator":{"name":"PhantomJS","version":"2.0.0"},"pages":[{"startedDateTime":"2015-07-14T19:01:40.795Z","id":"http://myapp.domain.com:8081/people/1","title":"John Smith - MyAppName","pageTimings":{"onLoad":2900}}],"entries":[{"startedDateTime":"2015-07-14T19:03:22.559Z","time":148,"request":{"method":"GET","url":"http://myapp.domain.com:8081/people/1","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Accept","value":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},{"name":"Cache-Control","value":"max-age=0"},{"name":"User-Agent","value":"Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.0.0 Safari/538.1"}],"queryString":[],"headersSize":-1,"bodySize":-1},"response":{"status":200,"statusText":"OK","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Date","value":"Tue, 14 Jul 2015 19:03:22 GMT"},{"name":"Server","value":"WSGIServer/0.1 Python/2.7.6"},{"name":"Vary","value":"Cookie"},{"name":"X-Frame-Options","value":"SAMEORIGIN"},{"name":"Content-Type","value":"text/html; charset=utf-8"},{"name":"Set-Cookie","value":"csrftoken=wFzWPTm9aVkGLtPuOCcc1tIs6ve5KosW; expires=Tue, 12-Jul-2016 19:03:22 GMT; Max-Age=31449600; Path=/"}],"redirectURL":"","headersSize":-1,"bodySize":5776,"content":{"size":5776,"mimeType":"text/html; charset=utf-8"}},"cache":{},"timings":{"blocked":0,"dns":-1,"connect":-1,"send":0,"wait":140,"receive":8,"ssl":-1},"pageref":"http://myapp.domain.com:8081/people/1"},{"startedDateTime":"2015-07-14T19:03:22.705Z","time":7,"request":{"method":"GET","url":"http://myapp.domain.com:8081/static/js/lib/bower_components/requirejs/require.js","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Accept","value":"*/*"},{"name":"Referer","value":"http://myapp.domain.com:8081/people/1"},{"name":"User-Agent","value":"Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.0.0 Safari/538.1"}],"queryString":[],"headersSize":-1,"bodySize":-1},"response":{"status":null,"statusText":"Error downloading http://myapp.domain.com:8081/static/js/lib/bower_components/requirejs/require.js - server replied: NOT FOUND","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Date","value":"Tue, 14 Jul 2015 19:03:22 GMT"},{"name":"Server","value":"WSGIServer/0.1 Python/2.7.6"},{"name":"X-Frame-Options","value":"SAMEORIGIN"},{"name":"Content-Type","value":"text/html"}],"redirectURL":"","headersSize":-1,"bodySize":125,"content":{"size":125,"mimeType":"text/html"}},"cache":{},"timings":{"blocked":0,"dns":-1,"connect":-1,"send":0,"wait":6,"receive":1,"ssl":-1},"pageref":"http://myapp.domain.com:8081/people/1"},{"startedDateTime":"2015-07-14T19:03:22.706Z","time":15,"request":{"method":"GET","url":"http://myapp.domain.com:8081/static/js/lib/bower_components/jquery-ui/themes/ui-lightness/jquery.ui.theme.css","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Accept","value":"text/css,*/*;q=0.1"},{"name":"Referer","value":"http://myapp.domain.com:8081/people/1"},{"name":"User-Agent","value":"Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.0.0 Safari/538.1"}],"queryString":[],"headersSize":-1,"bodySize":-1},"response":{"status":null,"statusText":"Error downloading http://myapp.domain.com:8081/static/js/lib/bower_components/jquery-ui/themes/ui-lightness/jquery.ui.theme.css - server replied: NOT FOUND","httpVersion":"HTTP/1.1","cookies":[],"headers":[{"name":"Date","value":"Tue, 14 Jul 2015 19:03:22 GMT"},{"name":"Server","value":"WSGIServer/0.1 Python/2.7.6"},{"name":"X-Frame-Options","value":"SAMEORIGIN"},{"name":"Content-Type","value":"text/html"}],"redirectURL":"","headersSize":-1,"bodySize":154,"content":{"size":154,"mimeType":"text/html"}},"cache":{},"timings":{"blocked":0,"dns":-1,"connect":-1,"send":0,"wait":15,"receive":0,"ssl":-1},"pageref":"http://myapp.domain.com:8081/people/1"}]}}',
    'level': 'INFO'
}]

So with that I found JS assets weren’t being compiled in my CI environment and was able to go ahead and fix it :)

Hopefully that’s useful to someone out there.