AOSP build hangs at the beginning of build

I am developing a system with the AOSP fork, rowboat. It was working until today, now I get this:

$ time make -j8 TARGET_PRODUCT=beagleboneblack OMAPES=4.x droid showcommands
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=4.2.2
TARGET_PRODUCT=beagleboneblack
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
HOST_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-3.13.0-29-generic-x86_64-with-LinuxMint-17-qiana
HOST_BUILD_TYPE=release
BUILD_ID=JDQ39
OUT_DIR=out
============================================
^CTraceback (most recent call last):
  File "build/tools/findleaves.py", line 98, in <module>
    main(sys.argv)
  File "build/tools/findleaves.py", line 92, in main
    results = list(set(perform_find(mindepth, prune, dirlist, filename)))
  File "build/tools/findleaves.py", line 31, in perform_find

real    7m19.311s
user    0m0.236s
for root, dirs, files in os.walk(rootdir, followlinks=True):
sys 0m0.104s
  File "/usr/lib/python2.7/os.py", line 294, in walk

atilla@atilla-vaiod$     for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):

snip(this goes on so many lines).

  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 284, in walk
    if isdir(join(top, name)):
  File "/usr/lib/python2.7/genericpath.py", line 41, in isdir
    st = os.stat(s)
KeyboardInterrupt
^C

Solution

I found the problem. For some reason, the output is suppressed when the script is called from makefile. I ran the script manually, and found some recursive symlinks. Removed them and it is working again.

EDIT: Here is how I diagnosed the problem.

The build starts from build/core/main.mk, which invokes build/tools/findleaves.py. This script searches the entire source tree for files called Android.mk and stitches them together to create a temporary, gargantuan makefile.

If you start a build and cancel it by pressing Ctrl-C in the first few seconds, you will see the familiar keyboard interrupt message from Python. When I did it, the Python stack dump was several pages tall, as the script searches the directory tree recursively.

What I did was:

I added a simple print statement into the script to see what directory it scans.
This is important: I invoked the script directly, instead of through the makefile. This way, I can see its output, which would otherwise consumed by the makefile. Printing the traces to stderr instead of stdout will probably work as well.

The problem was, I created some extra subdirectories in the kernel directory to "install" modules and firmware, to make it easy for me to copy later. The make modules-install command installs a symlink back to the kernel source tree, pointing where the modules come from. So the there is a file kernel/deploy/lib/modules/3.8.13/source which is a symlink back to the kernel/ directory.

The Python script follows symlinks and tries to scan a tree with infinite depth.