Getting "Return from incompatible pointer type" warning, but i don't see why

Question

I'm trying to make a maze generator in C, but i get a warning for some reason. If the answer has something to do with memory allocation, please explain pretty in depth because i have no idea on how that works.

#include <stdlib.h>
#include <stdio.h>

unsigned char ** gen(int w, int h){
    unsigned char maze[w+2][h+2];

    for(int i=0; i<w; i++){
        if(i==0 || i==w-1){
            for(int j=0; j<h; j++){
                maze[i][j]=1;
            }
        }else{
            for(int j=0; j<h; j++){
                if(j==0 || j==h-1){
                    maze[i][j]=1;
                }else{
                    maze[i][j]=0;
                }
            }
        }
    }
    return maze;
}

I have looked around on the internet but i have not found anything that works.

edit: fixed typo in code

Answer 1

You have two main problems here.

First you're trying to return an array of type unsigned char [w+2][h+2] from a function that expects the return type to be unsigned char **. These types are incompatible.

While an array does decay to a pointer to its first element, this only applied to the outermost dimension, not all of them. So an array of type unsigned char [w+2][h+2] actually decays to a pointer of type unsigned char (*)[h+2], i.e. a pointer to an array of usigned char of size h+2, and this type is not compatible with unsigned char **.

Second, you're attempting to return a pointer to a local variable, in this case the 2D array. That array's lifetime will end when the function returns, so attempting to use the returned pointer will trigger undefined behavior.

You can fix the local return issue by dynamically allocating space for the 2D array:

unsigned char (*maze)[h+2] = malloc((w+2) * sizeof *maze);

For the return type mismatch, you can change the function to return a pointer to an array:

unsigned char (*gen(int w, int h))[] {

And assign the return value to a pointer of the proper type:

unsigned char (*m)[h+2] = gen(w,h);

Note here that the function returns a pointer to an array of unspecified size. This is necessary because the size of the pointed-to array is not known at compile time. However it is valid code because it is allowed to convert between a pointer to an array of known size and an array of unspecified size.

On a side note, you have a bug at the start of the function:

for(int i=0; 1<w; i++){

This should be:

for(int i=0; i<w; i++){

And this does nothing:

maze[i][j]==0;

It should instead be:

maze[i][j]=0;