Encoded nanopb protocol buffer consumes more memory than expected

Question

I'm trying to convert the following c structure:

typedef struct _algo_stream_debug_pkt {
  uint16_t  src; 
  uint16_t  dest; 
  uint16_t  length; 
  uint16_t  crc; 
  uint8_t  command; 
  uint8_t  status; 
  uint16_t  sequence_num; 
  uint32_t  timestamp; 
  uint16_t  data_state; 
  uint16_t  algo_out_x1; 
  uint16_t  algo_out_x2; 
  uint16_t  algo_out_x2; 
  uint16_t  algo_interval; 
  uint16_t  debug_info[10]; 
} algo_stream_debug_pkt ;

to protocol buffer with the following .proto file:

message m_header {
    required int32 src = 1;
    required int32 dst = 2;
    required int32 len = 3;
    required int32 crc = 4;
}

message algo_stream_debug_pkt {
    required m_header        header          = 1;
    required int32  command                  = 2; 
    required int32  status                   = 3; 
    required int32  sequence_num             = 4;                  
    required int32  timestamp                = 5;          
    required int32  data_state               = 6;              
    required int32  algo_out_x1              = 7;  
    required int32  algo_out_x2              = 8;            
    required int32  algo_out_x3              = 9;        
    required int32  algo_interval            = 10;
    repeated int32  debug_info               = 11;
}

and the following .option file:

algo_stream_debug_pkt.debug_info max_count:10

The issue is that the encoded protocol buffer data of type algo_stream_debug_pkt uses approximately 58 bytes (when streamed to memory) for my use case. The actual c structure of algo_stream_debug_pkt uses just 48 bytes [ sizeof(algo_stream_debug_pkt) -> 48 ]. I'm wondering why the protocol buffer version consumes more space than the non - encoded c struct version given that the nanopb uses varints to represent integer types so that actual encoded version should be consuming lesser memory than the initially specified type.

Answer 1

The issue is that the encoded protocol buffer data of type algo_stream_debug_pkt uses approximately 58 bytes (when streamed to memory) for my use case. The actual c structure of algo_stream_debug_pkt uses just 48 bytes [ sizeof(algo_stream_debug_pkt) -> 48 ]

Each protobuf field is prefixed with the field tag number, which takes up at least one byte. That is 15 bytes spent for the field tags, and 2 bytes for the header submessage and debug_info array length.

On the other hand, some space is saved by encoding the values as variable length integers, which only takes one byte for values less than 128.

In general protobuf format is quite space-efficient, but it is not a compression format. The requirement of forward- and backward-compatibility in the protocol also causes some overhead.