我正在尝试通过阅读他们的教程并进行小的修改来学习 tensorflow。我遇到了一个错误，对他们的代码进行微小的更改会导致输出变为 nan。
他们的原始代码是这样的:
import numpy as np
import tensorflow as tf
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Model input and output
x = tf.placeholder(tf.float32)
linear_model = W * x + b
y = tf.placeholder(tf.float32)
# loss
loss = tf.reduce_sum(tf.square(linear_model - y)) # sum of the squares
# optimizer
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss)
# training data
x_train = [1,2,3,4]
y_train = [0,-1,-2,-3]
# training loop
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init) # reset values to wrong
for i in range(1000):
sess.run(train, {x:x_train, y:y_train})
# evaluate training accuracy
curr_W, curr_b, curr_loss = sess.run([W, b, loss], {x:x_train, y:y_train})
print("W: %s b: %s loss: %s"%(curr_W, curr_b, curr_loss))
这个的输出是:
>python linreg2.py
2017-07-22 22:19:41.409167: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:19:41.409311: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:19:41.412452: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:19:41.412556: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:19:41.412683: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:19:41.412826: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:19:41.412958: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:19:41.413086: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
W: [-0.9999969] b: [ 0.99999082] loss: 5.69997e-11
注意我每次运行时收到的所有消息，因为我使用 pip 安装，并没有自己编译。然而，它确实得到了正确的输出，W = -1 和 b = 1
我修改了代码，只是添加了 x_train 和 y_train 变量:
import numpy as np
import tensorflow as tf
# Model parameters
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
# Model input and output
x = tf.placeholder(tf.float32)
linear_model = W * x + b
y = tf.placeholder(tf.float32)
# loss
loss = tf.reduce_sum(tf.square(linear_model - y)) # sum of the squares
# optimizer
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss)
# training data
x_train = [1,2,3,4,5,6,7]
y_train = [0,-1,-2,-3,-4,-5,-6]
# training loop
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init) # reset values to wrong
for i in range(1000):
sess.run(train, {x:x_train, y:y_train})
# evaluate training accuracy
curr_W, curr_b, curr_loss = sess.run([W, b, loss], {x:x_train, y:y_train})
print("W: %s b: %s loss: %s"%(curr_W, curr_b, curr_loss))
这是这段新代码的输出:
2017-07-22 22:23:13.129983: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:23:13.130125: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:23:13.130853: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:23:13.130986: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:23:13.131126: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:23:13.131234: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:23:13.132178: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-22 22:23:13.132874: W c:\tf_jenkins\home\workspace\release-win\m\windows\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
W: [ nan] b: [ nan] loss: nan
我真的不知道为什么扩展训练数据会导致这种情况发生。有什么我想念的吗？
此外，我完全不确定如何调试 TF 中的内容，比如让值在循环中进行增量打印并更改变量。仅仅打印出变量似乎不起作用。我想知道，以便将来可以自己调试这些东西!
请您参考如下方法:
欢迎来到超参数调整的美妙世界。您可以尝试以下方法，首先不是在最后提供一些输出，您还可以在 for 循环中打印一些输出，这可能会变成:
for i in range(1000):
curr_W, curr_b, curr_loss,_ = sess.run([W, b, loss, train], {x:x_train, y:y_train})
print("Iteration %d W: %s b: %s loss: %s"%(i, curr_W, curr_b, curr_loss))
如果你运行这个然后输出看起来像:
Iteration 0 W: [-2.61199999] b: [-0.84599996] loss: 153.79
Iteration 1 W: [ 2.93535995] b: [ 0.31516004] loss: 554.292
Iteration 2 W: [-7.70013809] b: [-1.79276371] loss: 2020.55
Iteration 3 W: [ 12.6241951] b: [ 2.35030031] loss: 7387.32
Iteration 4 W: [-26.27972031] b: [-5.46829081] loss: 27029.6
Iteration 5 W: [ 48.12573624] b: [ 9.59391212] loss: 98918.8
Iteration 6 W: [-94.23892212] b: [-19.11964607] loss: 362027.0
Iteration 7 W: [ 178.09707642] b: [ 35.9108963] loss: 1.32498e+06
Iteration 8 W: [-342.92483521] b: [-69.27098846] loss: 4.84928e+06
Iteration 9 W: [ 653.81640625] b: [ 132.04486084] loss: 1.77479e+07
Iteration 10 W: [-1253.05480957] b: [-252.99859619] loss: 6.49554e+07
...
Iteration 60 W: [ -1.52910250e+17] b: [ -3.08788499e+16] loss: 9.6847e+35
Iteration 61 W: [ 2.92530566e+17] b: [ 5.90739251e+16] loss: 3.54451e+36
Iteration 62 W: [ -5.59636369e+17] b: [ -1.13013526e+17] loss: 1.29725e+37
Iteration 63 W: [ 1.07063302e+18] b: [ 2.16204754e+17] loss: 4.74782e+37
Iteration 64 W: [ -2.04821397e+18] b: [ -4.13618407e+17] loss: 1.73766e+38
Iteration 65 W: [ 3.91841178e+18] b: [ 7.91287870e+17] loss: inf
Iteration 66 W: [ -7.49626247e+18] b: [ -1.51380280e+18] loss: inf
Iteration 67 W: [ 1.43410016e+19] b: [ 2.89603611e+18] loss: inf
Iteration 68 W: [ -2.74355815e+19] b: [ -5.54036982e+18] loss: inf
Iteration 69 W: [ 5.24866609e+19] b: [ 1.05992074e+19] loss: inf
...
Iteration 126 W: [ -6.01072457e+35] b: [ -1.21381189e+35] loss: inf
Iteration 127 W: [ 1.14990384e+36] b: [ 2.32212753e+35] loss: inf
Iteration 128 W: [ -2.19986564e+36] b: [ -4.44243161e+35] loss: inf
Iteration 129 W: [ inf] b: [ 8.49875587e+35] loss: inf
Iteration 130 W: [ nan] b: [-inf] loss: inf
Iteration 131 W: [ nan] b: [ nan] loss: nan
Iteration 132 W: [ nan] b: [ nan] loss: nan
此时您应该能够看到 W 和 b 的值正在积极更新，而不是减少您的损失实际上正在增加，并且非常快地接近无穷大。这反过来意味着你的学习率已经偏离了。如果将学习率除以 10 并将其设置为 0.001，则最终结果为:
W: [-0.97952145] b: [ 0.8985914] loss: 0.0144026
这表明您的模型尚未收敛(同时查看之前的输出，理想情况下您创建损失图。学习率设置为 0.05 的下一个实验给出:
W: [-0.99999958] b: [ 0.99999791] loss: 6.48015e-12
因此得出结论:
- 尝试从 sess.run()(或某些张量的 eval())中提取中间结果并查看模型的学习情况。
- 为了乐趣和 yield 而调整超参数。
注意:此时您仍在使用具有固定学习率的“简单”梯度下降，但也有自动调整学习率的优化器。优化器(及其参数)的选择也是其他超参数。
