podman unable to build image from Dockerfile error creating overlay mount

I am getting error as error creating overlay mount to /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged

Please find debug info below.

OS

root@cks-master:/vagrant/files# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:        18.04
Codename:       bionic

Dockerfile

# cat ch5Dockerfile 
FROM bash
CMD ["ping", "killer.sh"]

Build log

root@cks-master:/vagrant/files# podman build -t simple -f ch5Dockerfile .
STEP 1/2: FROM bash
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged: invalid argument 
Error: error mounting new container: error mounting build container "6c0f88a6da54d713e18283e16521385fff736bc1a1072938fddfc6be4b3d43cc": error creating overlay mount to /var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/BNREFG6CRAAHJ7VSYG3EUXV5UO:/var/lib/containers/storage/overlay/l/MDSWZVRVZNCOW75JF32K6D4QQC:/var/lib/containers/storage/overlay/l/4G3NS52LYHWPTKA4FURHLYMAPF,upperdir=/var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/diff,workdir=/var/lib/containers/storage/overlay/7a617fad39ce9178c810e29aaef4af73647d8e35ae0969483059441c1c4ee9cd/work": invalid argument
root@cks-master:/vagrant/files#

podman version

root@cks-master:/vagrant/files# podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.15.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64
root@cks-master:/vagrant/files#

Solution

Issue resolved.

I also posted the issue in podman room at https://app.element.io/#/room/#podman:fedoraproject.org

I am suggested with podman reset(command trace is below), then its complained about storage.conf, I removed that file and did reset again. Then its worked.

I still wonder whats inside of storage.conf causing this issue but I deleted before looking into it. Finally its worked and I am able to continue. Hope it helps.

Note: Post the deleting storage.conf file and podman reset, I tried with docker build as well just to check if docker has any dependency over storage.conf file, but none, docker build also executed successfully( command trace below)

root@cks-master:~# podman system reset -f
A storage.conf file exists at /etc/containers/storage.conf
You should remove this file if you did not modified the configuration.

root@cks-master:~# rm /etc/containers/storage.conf

root@cks-master:~# podman system reset -f

root@cks-master:~# podman build -t simple -f /vagrant/files/ch5Dockerfile .
STEP 1/2: FROM bash
Resolving "bash" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/bash:latest...
Getting image source signatures
Copying blob 9621f1afde84 done  
Copying blob 1dd831616e40 done  
Copying blob fd6cd28e0879 done  
Copying config 8b332999f6 done  
Writing manifest to image destination
Storing signatures
STEP 2/2: CMD ["ping", "killer.sh"]
COMMIT simple
--> cd1407a69ea
Successfully tagged localhost/simple:latest
cd1407a69ea490496d6635700958f2b5fcf2b1d01f8dd218dea0f83187e55872

root@cks-master:~# podman run --name simple simple
PING killer.sh (35.227.196.29): 56 data bytes
64 bytes from 35.227.196.29: seq=0 ttl=42 time=15.689 ms
64 bytes from 35.227.196.29: seq=1 ttl=42 time=14.662 ms
64 bytes from 35.227.196.29: seq=2 ttl=42 time=15.161 ms
^C
--- killer.sh ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 14.662/15.170/15.689 ms



root@cks-master:~# docker build -t simple -f /vagrant/files/ch5Dockerfile .
Sending build context to Docker daemon  3.141MB
Step 1/2 : FROM bash
latest: Pulling from library/bash
9621f1afde84: Pull complete 
1dd831616e40: Pull complete 
fd6cd28e0879: Pull complete 
Digest: sha256:e4624241e953934fc4c396217253d8322ebda53be3b1863cd7795541d168034f
Status: Downloaded newer image for bash:latest
 ---> 8b332999f684
Step 2/2 : CMD ["ping", "killer.sh"]
 ---> Running in 306963a83d1c
Removing intermediate container 306963a83d1c
 ---> 51dee555fd57
Successfully built 51dee555fd57
Successfully tagged simple:latest
root@cks-master:~# ^C
root@cks-master:~#