Sometimes you’ll get a segmentation fault in Python and your process will crash, this is due to a C module attempting to access memory beyond reach.
Output in this case will be very limited:
Segmentation fault
To get a full traceback we need to enable the Python faulthandler module.
First add the following to the top of your module.
import faulthandler; faulthandler.enable()
Then re-run your program with the faulthandler startup flag.
Passed as an argument.
# pass as an argument python -Xfaulthandler my_program.py # Or as an environment variable. PYTHONFAULTHANDLER=1 python my_program.py
Now you will get a detailed traceback.
Fatal Python error: Segmentation fault Current thread 0x00007f0a49caa700 (most recent call first): File "", line 219 in _call_with_frames_removed File "", line 922 in create_module File "", line 571 in module_from_spec File "", line 658 in _load_unlocked File "", line 684 in _load File "/usr/local/lib/python3.6/imp.py", line 343 in load_dynamic File "/usr/local/lib/python3.6/imp.py", line 243 in load_module File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24 in swig_import_helper File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1023 in _handle_fromlist File "/usr/local/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/usr/local/lib/python3.6/site-packages/tensorflow/__init__.py", line 24 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/usr/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 5 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/usr/local/lib/python3.6/site-packages/keras/backend/__init__.py", line 83 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1023 in _handle_fromlist File "/usr/local/lib/python3.6/site-packages/keras/utils/conv_utils.py", line 9 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1023 in _handle_fromlist File "/usr/local/lib/python3.6/site-packages/keras/utils/__init__.py", line 6 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1023 in _handle_fromlist File "/usr/local/lib/python3.6/site-packages/keras/__init__.py", line 3 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 941 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/opt/app/ai_platform/forecasting.py", line 7 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/opt/app/ai_platform/customer_operations.py", line 12 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "/opt/app/app.py", line 13 in File "", line 219 in _call_with_frames_removed File "", line 678 in exec_module File "", line 665 in _load_unlocked File "", line 955 in _find_and_load_unlocked File "", line 971 in _find_and_load File "", line 994 in _gcd_import File "/usr/local/lib/python3.6/importlib/__init__.py", line 126 in import_module File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101 in import_from_cwd File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56 in symbol_by_name File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 506 in symbol_by_name File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 355 in find_app File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 503 in find_app File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 481 in setup_app_from_commandline File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 279 in execute_from_commandline ... Segmentation fault
Happy debugging!
thanks for your information
Hi Richard,
Thanks for this information.
However, were you able to debug this particular issue that you were facing?
I’ve the segmentation fault thrown at the same place but unable to figure out a fix.