I am running a python program which uses VGG16 neural network, through the keras package, to classify images of cats and dogs, from the Kaggle database. To do that, I am using the standard terminal command: python program.py > output.txt. I have also tried out the other variants, python program.py &> output.txt, or, the tee command, python program.py |& tee output.txt, but it does not seem to be working. For the first command, my text file contains just:
Using TensorFlow backend.
2017-05-31 13:39:34.218034: W tensorflow/core/platform/cpu_feature_guard.cc:45]
The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are
available on your machine and could speed up CPU computations.
2017-05-31 13:39:34.226941: W tensorflow/core/platform/cpu_feature_guard.cc:45]
The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are
available on your machine and could speed up CPU computations.
but the code has a lot of print statements! The expected content of the output.txt file is (only the first 4-5 lines of the terminal output shown):
Using TensorFlow backend.
Defining all the path!
All paths defined!
Getting mean RGB and creating labels!
which is displayed when I just type python program.py. The part:
2017-05-31 13:39:34.218034: W tensorflow/core/platform/cpu_feature_guard.cc:45]
The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are
available on your machine and could speed up CPU computations.
2017-05-31 13:39:34.226941: W tensorflow/core/platform/cpu_feature_guard.cc:45]
The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are
available on your machine and could speed up CPU computations.
part comes much later in the terminal output. I am putting my code here for reference, but it is 204 lines long:
import keras
from keras.models import Sequential, Model
from keras.layers import Flatten, Dense, Dropout, Input, Activation
from keras.layers.convolutional import Conv2D, MaxPooling2D, ZeroPadding2D
from keras.layers.merge import Add
from keras.optimizers import SGD, Adam
import cv2, numpy as np
import glob
import csv
####################
## VGG16 Function ##
####################
def VGG_16(weights_path=None, classes=2):
######################################
## Input: 3x224x224 sized RGB Input ##
######################################
inputs = Input(shape=(3,224,224))
layer = 0
#############
## Block 1 ##
#############
x = Conv2D(64, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block1_conv1')(inputs)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = Conv2D(64, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block1_conv2')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)
#############
## Block 2 ##
#############
x = Conv2D(128, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block2_conv1')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = Conv2D(128, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block2_conv2')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = MaxPooling2D((2, 2), strides=(2, 2), name='block2_pool')(x)
#############
## Block 3 ##
#############
x = Conv2D(256, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block3_conv1')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = Conv2D(256, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block3_conv2')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = Conv2D(256, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block3_conv3')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = MaxPooling2D((2, 2), strides=(2, 2), name='block3_pool')(x)
#############
## Block 4 ##
#############
x = Conv2D(512, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block4_conv1')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = Conv2D(512, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block4_conv2')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = Conv2D(512, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block4_conv3')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = MaxPooling2D((2, 2), strides=(2, 2), name='block4_pool')(x)
#############
## Block 5 ##
#############
x = Conv2D(512, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block5_conv1')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = Conv2D(512, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block5_conv2')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
x = Conv2D(512, (3, 3), data_format='channels_first', activation='relu', padding='same', name='block5_conv3')(x)
layer += 1
print ('Output shape for Layer ' +str(layer)+ ', is, ' +str(x.get_shape()))
out = MaxPooling2D((2, 2), strides=(2, 2), name='block5_pool')(x)
###############
## Top layer ##
###############
out = Flatten(name='flatten')(out)
out = Dense(4096, activation='relu', name='fc1')(out)
out = Dropout(0.5)(out)
out = Dense(4096, activation='relu', name='fc2')(out)
out = Dropout(0.5)(out)
out = Dense(classes, activation='softmax', name='predictions')(out)
if weights_path:
model.load_weights(weights_path)
model = Model(inputs, out, name='vgg-16')
return model
###################
## Main Function ##
###################
if __name__ == "__main__":
################################################
## Get all the training and the testing paths ##
################################################
print('Defining all the path!\n')
cat_path = "./train/cat.*.jpg"
dog_path = "./train/dog.*.jpg"
train_path = "./train/*.jpg"
test_path = "./test1/*.jpg"
Mean_RGB = []
x_train = []
y_train = []
x_test = []
print('All paths defined!\n')
########################################################################
## Get training and testng data sizes, to find the average RGB values ##
########################################################################
print('Getting mean RGB and creating labels!\n')
for file in glob.glob(cat_path): # To get the sizes of all the cat images
im = cv2.resize(cv2.imread(file), (224, 224)).astype(np.float32)
im = np.mean(im, axis=(0,1))
Mean_RGB.append(tuple(im))
y_train.append(0)
for file in glob.glob(dog_path): # To get the sizes of all the dog images
im = cv2.resize(cv2.imread(file), (224, 224)).astype(np.float32)
im = np.mean(im, axis=(0,1))
Mean_RGB.append(tuple(im))
y_train.append(1)
y_train = np.array(y_train)
Mean_RGB = tuple(np.mean(Mean_RGB, axis=0))
print('Got mean RGB and created labels!\n')
#########################################################################
## Load the training and testing images, after subtracting average RGB ##
#########################################################################
print('Loading images as numpy arrays!\n')
for file in glob.glob(train_path):
im = cv2.resize(cv2.imread(file), (224, 224)).astype(np.float32)
im_r = im-Mean_RGB
im_r = im_r.transpose((2,0,1))
#im_r = np.expand_dims(im_r, axis=0)
x_train.append(im_r)
y_train = y_train.reshape((-1,1))
y_train = keras.utils.to_categorical(y_train, num_classes=2)
x_train = np.array(x_train)
for file in glob.glob(test_path):
im = cv2.resize(cv2.imread(file), (224, 224)).astype(np.float32)
im_r = im-Mean_RGB
im_r = im_r.transpose((2,0,1))
#im_r = np.expand_dims(im_r, axis=0)
x_test.append(im_r)
x_test = np.array(x_test)
print('All images loaded!\n')
##############################
## Train and test the model ##
##############################
print('Creating Neural Net!\n')
model = VGG_16()
print('\nNeural Net created!\n')
adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(optimizer=adam, loss='binary_crossentropy', metrics=['accuracy'])
print('Training Neural Net!\n')
### Generating validation data split in training sample
model.fit(x_train, y_train, batch_size=500, epochs=25, validation_split=0.2, shuffle=True)
print('Neural Net trained!\n')
print('Evaluating model on the training images!\n')
score = model.evaluate(x_train, y_train, batch_size=500, verbose=1)
print('Model score on training data: ' +str(score)+ '\n')
print('Predicting class of test images!\n')
pred = model.predict(x_test, batch_size=1, verbose=1)
prediction = np.argmax(pred, axis = 1)
print('Predictions done!\n')
result = []
print('Creating output CSV file!\n')
result.append(['id', 'label'])
for i in range(0,len(prediction)):
result.append([i+1,prediction[i]])
with open("cat-dog-output.csv","wb") as f:
writer = csv.writer(f)
writer.writerows(result)
print('Created output CSV file!\n')
print('Saving model parameters!\n')
model.save('vgg16-sim-conn.h5')
model.save_weights('vgg16-sim-conn-weights.h5')
print('Model saved!\n')
I do not know what is really going on and any help in this matter will be deeply appreciated!
After some amount of tinkering with the python command, and the help, python -h I found that there is an option -u for unbuffered output. I tried it out, python -u program.py > tee output.txt and it worked perfectly.
I had posted my question in Ask Ubuntu as well and Steven D. suggested the same solution. His answer also redirected me to a similar question in StackOverflow.