How to debug a “Segmentation fault” in Python

Sometimes you’ll get a segmentation fault in Python and your process will crash, this is due to a C module attempting to access memory beyond reach.

Output in this case will be very limited:

Segmentation fault

To get a full traceback we need to enable the Python faulthandler module.

First add the following to the top of your module.

import faulthandler; faulthandler.enable()

Then re-run your program with the faulthandler startup flag.

Passed as an argument.

# pass as an argument
python -Xfaulthandler my_program.py

# Or as an environment variable.
PYTHONFAULTHANDLER=1 python my_program.py

Now you will get a detailed traceback.

Fatal Python error: Segmentation fault
Current thread 0x00007f0a49caa700 (most recent call first):
File "", line 219 in _call_with_frames_removed
File "", line 922 in create_module
File "", line 571 in module_from_spec
File "", line 658 in _load_unlocked
File "", line 684 in _load
File "/usr/local/lib/python3.6/imp.py", line 343 in load_dynamic
File "/usr/local/lib/python3.6/imp.py", line 243 in load_module
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24 in swig_import_helper
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/__init__.py", line 24 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 5 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/__init__.py", line 83 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/conv_utils.py", line 9 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/__init__.py", line 6 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/__init__.py", line 3 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 941 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/forecasting.py", line 7 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/customer_operations.py", line 12 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/app.py", line 13 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 994 in _gcd_import
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126 in import_module
File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101 in import_from_cwd
File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 506 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 355 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 503 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 481 in setup_app_from_commandline
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279 in execute_from_commandline
...
Segmentation fault

Happy debugging!

3 thoughts on “How to debug a “Segmentation fault” in Python

  1. Nisith Singh

    Hi Richard,

    Thanks for this information.
    However, were you able to debug this particular issue that you were facing?
    I’ve the segmentation fault thrown at the same place but unable to figure out a fix.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *