In the TheAdmin theme in OrchardCore.Themes, I see in Views folder Layout.cshtml and Layout-Login.cshtml.
In the TheAgency theme, I see Layout.liquid in views folder.
I tried to change my Layout.liquid in my views folder in my newly created theme to Layout.cshtml and when I run it, it still works.
So, what is the difference between liquid and cshtml file in themes in Orchard Core, and when should I use one over the other?
They are two different view engines that you can use. Razor/cshtml is more familiar to ASP.NET developers, but its views are compiled. Liquid is a very well-known templating language in the JavaScript world, and it has the huge advantage for Orchard that its views are not compiled. That enables storing them in the database, changing them without restarting or triggering compilation, etc.
As a module author, you have a choice, and should use the one you're most comfortable with. Sometimes however, like if the view must be dynamically created for whatever reason, Liquid is the only choice.