Sometimes you’ll get a segmentation fault in Python and your process will crash, this is due to a C module attempting to access memory beyond reach.
Output in this case will be very limited:
Segmentation fault
To get a full traceback we need to enable the Python faulthandler module.
First add the following to the top of your module.
import faulthandler; faulthandler.enable()
Then re-run your program with the faulthandler startup flag.
Passed as an argument.
# pass as an argument python -Xfaulthandler my_program.py # Or as an environment variable. PYTHONFAULTHANDLER=1 python my_program.py
Now you will get a detailed traceback.
Fatal Python error: Segmentation fault Current thread 0x00007f0a49caa700 (most recent call first): File "", line 219 in _call_with_frames_removed File "", line 922 in create_module File "", line 571 in module_from_spec File "", line 658 in _load_unlocked File "", line 684 in _load File "/usr/local/lib/python3.6/imp.py", line 343 in load_dynamic File "/usr/local/lib/python3.6/imp.py", line 243 in load_module File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24 in swig_import_helper File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1023 in _handle_fromlist File "/usr/local/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/usr/local/lib/python3.6/site-packages/tensorflow/__init__.py", line 24 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 5 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/usr/local/lib/python3.6/site-packages/keras/backend/__init__.py", line 83 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1023 in _handle_fromlist File "/usr/local/lib/python3.6/site-packages/keras/utils/conv_utils.py", line 9 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1023 in _handle_fromlist File "/usr/local/lib/python3.6/site-packages/keras/utils/__init__.py", line 6 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1023 in _handle_fromlist File "/usr/local/lib/python3.6/site-packages/keras/__init__.py", line 3 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 941 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/opt/app/ai_platform/forecasting.py", line 7 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/opt/app/ai_platform/customer_operations.py", line 12 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/opt/app/app.py", line 13 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 994 in _gcd_import File "/usr/local/lib/python3.6/importlib/__init__.py", line 126 in import_module File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101 in import_from_cwd File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56 in symbol_by_name File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 506 in symbol_by_name File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 355 in find_app File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 503 in find_app File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 481 in setup_app_from_commandline File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279 in execute_from_commandline ... Segmentation fault
Happy debugging!