How should strace be used?

A colleague once told me that the last option when everything has failed to debug on Linux was to use strace.

I tried to learn the science behind this strange tool, but I am not a system admin guru and I didn’t really get results.

So,

What is it exactly and what does it do?
How and in which cases should it be used?
How should the output be understood and processed?

In brief, in simple words, how does this stuff work?

Solution

Strace Overview
strace can be seen as a light weight debugger. It allows a programmer / user to quickly find out how a program is interacting with the OS. It does this by monitoring system calls and signals.

Uses
Good for when you don't have source code or don't want to be bothered to really go through it.
Also, useful for your own code if you don't feel like opening up GDB, but are just interested in understanding external interaction.

A good little introduction
Here is a gentle introduction to using strace to debug process hangs: strace introduction