首页 > 编程 > Python > 正文

tensorflow训练中出现nan问题的解决

2020-01-04 15:52:44
字体:
来源:转载
供稿:网友

深度学习中对于网络的训练是参数更新的过程,需要注意一种情况就是输入数据未做归一化时,如果前向传播结果已经是[0,0,0,1,0,0,0,0]这种形式,而真实结果是[1,0,0,0,0,0,0,0,0],此时由于得出的结论不惧有概率性,而是错误的估计值,此时反向传播会使得权重和偏置值变的无穷大,导致数据溢出,也就出现了nan的问题。

解决办法:

1、对输入数据进行归一化处理,如将输入的图片数据除以255将其转化成0-1之间的数据;

2、对于层数较多的情况,各层都做batch_nomorlization;

3、对设置Weights权重使用tf.truncated_normal(0, 0.01, [3,3,1,64])生成,同时值的均值为0,方差要小一些;

4、激活函数可以使用tanh;

5、减小学习率lr。

实例:

import tensorflow as tffrom tensorflow.examples.tutorials.mnist import input_datamnist = input_data.read_data_sets('data',one_hot = True)def add_layer(input_data,in_size, out_size,activation_function=None):  Weights = tf.Variable(tf.random_normal([in_size,out_size]))  Biases = tf.Variable(tf.zeros([1, out_size])+0.1)  Wx_plus_b = tf.add(tf.matmul(input_data, Weights), Biases)  if activation_function==None:    outputs = Wx_plus_b  else:    outputs = activation_function(Wx_plus_b)  #return outputs#, Weights  return {'outdata':outputs, 'w':Weights}def get_accuracy(t_y):#  global l1#  accu = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(l1['outdata'],1),tf.argmax(t_y,1)), dtype = tf.float32))  global prediction  accu = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(prediction['outdata'],1),tf.argmax(t_y,1)), dtype = tf.float32))  return accuX = tf.placeholder(tf.float32, [None, 784])Y = tf.placeholder(tf.float32, [None, 10])#l1 = add_layer(X, 784, 10, tf.nn.softmax)#cross_entropy = tf.reduce_mean(-tf.reduce_sum(Y*tf.log(l1['outdata']), reduction_indices= [1]))#l1 = add_layer(X, 784, 1024, tf.nn.relu)l1 = add_layer(X, 784, 1024, None)prediction = add_layer(l1['outdata'], 1024, 10, tf.nn.softmax)cross_entropy = tf.reduce_mean(-tf.reduce_sum(Y*tf.log(prediction['outdata']), reduction_indices= [1]))optimizer = tf.train.GradientDescentOptimizer(0.000001)train = optimizer.minimize(cross_entropy)newW = tf.Variable(tf.random_normal([1024,10]))newOut = tf.matmul(l1['outdata'],newW)newSoftMax = tf.nn.softmax(newOut)init = tf.global_variables_initializer()with tf.Session() as sess:  sess.run(init)  #print(sess.run(l1_Weights))  for i in range(2):    X_train, y_train = mnist.train.next_batch(1)    X_train = X_train/255  #需要进行归一化处理    #print(sess.run(l1['w'],feed_dict={X:X_train}))    #print(sess.run(prediction['w'],feed_dict={X:X_train, Y:y_train}))    #print(sess.run(l1['outdata'],feed_dict={X:X_train, Y:y_train}).shape)    print(sess.run(prediction['outdata'],feed_dict={X:X_train, Y:y_train}))    print(sess.run(newOut, feed_dict={X:X_train}))    print(sess.run(newSoftMax, feed_dict={X:X_train}))    print(y_train)    #print(sess.run(l1['outdata'], feed_dict={X:X_train}))    sess.run(train, feed_dict={X:X_train, Y:y_train})    if i%100 == 0:      #print(sess.run(cross_entropy, feed_dict={X:X_train, Y:y_train}))      accuracy = get_accuracy(mnist.test.labels)      print(sess.run(accuracy,feed_dict={X:mnist.test.images}))        #if i%100==0:    #print(sess.run(prediction, feed_dict={X:X_train}))    #print(sess.run(cross_entropy, feed_dict={X:X_train,Y:y_train}))

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持VEVB武林网。


注:相关教程知识阅读请移步到python教程频道。
发表评论 共有条评论
用户名: 密码:
验证码: 匿名发表