How to choose the best layers for CNN and improve the precision?

Question

I am building a food recognition problem using CNN model. My data consists of 35 labels and 1017 images of size 224, I divide the train and valid data, the result is 824 train and 193 valid. I build structure for classes like below code. However, after more than 100 epochs, I ran the test dataset with only 2.85% prediction scores and the f1_score, precision, and recall indexes only appeared in 1-2 labels. How can I improve my prediction score as well as f1_score, precision, recall

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(224,224,3)),
    tf.keras.layers.Conv2D(32, (3,3), activation='relu',padding='same'),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Conv2D(64, (3,3), activation='relu',padding='same'),
    tf.keras.layers.Conv2D(64, (3,3), activation='relu',padding='same'),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(35, activation='softmax'),
])

model.compile(optimizer='adam',
              loss='categorical_crossentropy', 
              metrics=['accuracy'])

model.summary()

Answer 1

The most important thing is you need more data, on the available data if you are training the model for 100 epochs , it will definitely overfit for the training samples.

1. collect more data | annotate more data manually
2. As you have 35 classes, you should have balanced dataset for all the classes.
3. start with less convolution layers and FCL , gradually increase the layers and their parameters and save results for each epoch
4. With the number of classes you mention you will also require good amount of test data fro accurate results, and the image context might be different in the train and test samples.