Sometimes you’ll get a segmentation fault in Python and your process will crash, this is due to a C module attempting to access memory beyond reach.

Output in this case will be very limited:

Segmentation fault

To get a full traceback we need to enable the Python faulthandler module.

First add the following to the top of your module.

import faulthandler; faulthandler.enable()

Then re-run your program with the faulthandler startup flag.

Passed as an argument.

# pass as an argument
python -Xfaulthandler my_program.py

# Or as an environment variable.
PYTHONFAULTHANDLER=1 python my_program.py

Now you will get a detailed traceback.

Fatal Python error: Segmentation fault
Current thread 0x00007f0a49caa700 (most recent call first):
File "", line 219 in _call_with_frames_removed
File "", line 922 in create_module
File "", line 571 in module_from_spec
File "", line 658 in _load_unlocked
File "", line 684 in _load
File "/usr/local/lib/python3.6/imp.py", line 343 in load_dynamic
File "/usr/local/lib/python3.6/imp.py", line 243 in load_module
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24 in swig_import_helper
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/tensorflow/__init__.py", line 24 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 5 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/usr/local/lib/python3.6/site-packages/keras/backend/__init__.py", line 83 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/conv_utils.py", line 9 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/utils/__init__.py", line 6 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 1023 in _handle_fromlist
File "/usr/local/lib/python3.6/site-packages/keras/__init__.py", line 3 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 219 in _call_with_frames_removed
File "", line 941 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/forecasting.py", line 7 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/ai_platform/customer_operations.py", line 12 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "/opt/app/app.py", line 13 in
File "", line 219 in _call_with_frames_removed
File "", line 678 in exec_module
File "", line 665 in _load_unlocked
File "", line 955 in _find_and_load_unlocked
File "", line 971 in _find_and_load
File "", line 994 in _gcd_import
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126 in import_module
File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101 in import_from_cwd
File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 506 in symbol_by_name
File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 355 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 503 in find_app
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 481 in setup_app_from_commandline
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279 in execute_from_commandline
...
Segmentation fault

Happy debugging!