Skip to main content
 首页 » 编程设计

python之CUDNN_STATUS_ALLOC_FAILED 导致 Tensorflow 崩溃

2025年05月04日20mq0036

我在网上搜索了几个小时没有结果,所以想在这里问一下。

我正尝试按照 Sentdex 的教程制作一辆自动驾驶汽车,但在运行模型时,我遇到了一堆 fatal error 。我在网上搜索了解决方案,很多人似乎都有同样的问题。但是,我找到的所有解决方案(包括 this Stack-post)都不适合我。

这是我的软件:

  • Tensorflow:1.5,GPU 版本
  • CUDA:9.0,有补丁
  • CUDnn:7
  • Windows 10 专业版
  • python 3.6

硬件:

  • Nvidia 1070ti,带最新驱动
  • 英特尔 i5 7600K

这是崩溃日志:

2018-02-04 16:29:33.606903: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-02-04 16:29:33.608872: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-02-04 16:29:33.609308: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-02-04 16:29:35.145249: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED 2018-02-04 16:29:35.145563: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM 2018-02-04 16:29:35.149896: F C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms)

这是我的代码:

 import tensorflow as tf 
    import numpy as np 
    import cv2 
    import time 
    from PIL import ImageGrab 
    from getkeys import key_check 
    from alexnet import alexnet 
    import os 
    from sendKeys import PressKey, ReleaseKey, W,A,S,D,Sp 
 
    import random 
 
    WIDTH = 80 
    HEIGHT = 60 
    LR = 1e-3 
    EPOCHS = 10 
    MODEL_NAME = 'DiRT-AI-Driver-{}-{}-{}-epochs.model'.format(LR, 'alexnetv2', EPOCHS) 
 
    def straight(): 
        PressKey(W) 
        ReleaseKey(A) 
        ReleaseKey(S) 
        ReleaseKey(D) 
        ReleaseKey(Sp) 
    def left(): 
        PressKey(A) 
        ReleaseKey(W) 
        ReleaseKey(S) 
        ReleaseKey(D) 
        ReleaseKey(Sp) 
    def right(): 
        PressKey(D) 
        ReleaseKey(A) 
        ReleaseKey(S) 
        ReleaseKey(W) 
        ReleaseKey(Sp) 
    def brake(): 
        PressKey(S) 
        ReleaseKey(A) 
        ReleaseKey(W) 
        ReleaseKey(D) 
        ReleaseKey(Sp) 
    def handbrake(): 
        PressKey(Sp) 
        ReleaseKey(A) 
        ReleaseKey(S) 
        ReleaseKey(D) 
        ReleaseKey(W) 
 
    model = alexnet(WIDTH, HEIGHT, LR) 
    model.load(MODEL_NAME) 
 
 
    def main(): 
        last_time = time.time() 
        for i in list(range(4))[::-1]: 
            print(i+1) 
            time.sleep(1) 
 
 
    paused = False 
    while(True): 
            if not paused: 
                screen = np.array(ImageGrab.grab(bbox=(0,40,1024,768))) 
                screen = cv2.cvtColor(screen,cv2.COLOR_BGR2GRAY) 
                screen = cv2.resize(screen,(80,60)) 
                print('Loop took {} seconds'.format(time.time()-last_time)) 
                last_time = time.time() 
                print('took time') 
                prediction = model.predict([screen.reshape(WIDTH,HEIGHT,1)])[0] 
                print('predicted') 
                moves = list(np.around(prediction)) 
                print('got moves') 
                print(moves,prediction) 
 
                if moves == [1,0,0,0,0]: 
                    straight() 
                elif moves == [0,1,0,0,0]: 
                    left() 
                elif moves == [0,0,1,0,0]: 
                    brake() 
                elif moves == [0,0,0,1,0]: 
                    right() 
                elif moves == [0,0,0,0,1]: 
                    handbrake() 
 
            keys = key_check() 
 
            if 'T' in keys: 
                if paused: 
                    pased = False 
                    time.sleep(1) 
                else: 
                    paused = True 
                    ReleaseKey(W) 
                    ReleaseKey(A) 
                    ReleaseKey(S) 
                    ReleaseKey(D) 
                    ReleaseKey(Sp) 
                    time.sleep(1) 
 
 
main() 

我发现导致 python 崩溃并产生前三个错误的行是这一行:

  • prediction = model.predict([screen.reshape(WIDTH,HEIGHT,1)])[0]

运行代码时,CPU 占用率高达 100%,表明出现严重问题。 GPU 达到约 40-50%

我已经尝试过 Tensorflow 1.2 和 1.3,以及 CUDA 8,但都没有用。安装 CUDA 时,我没有安装特定的驱动程序,因为它们对我的 GPU 来说太旧了。也尝试了不同的 CUDnn,但效果不佳。

请您参考如下方法:

在我的例子中,问题的发生是因为另一个导入了 tensorflow 的 python 控制台正在运行。关闭它解决了问题。

我有 Windows 10,主要错误是:

failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED