Search code examples
scons

When should I use scons scanner vs parseDepends and Depends

My apologies for this long question.

The following is what I would like to do with scons

  • Build a java component before a grammar files component (it is a voice-recognition project). My grammar files component needs the java component for his own compilation.
  • Using multi-processing (-j)

What is working so far :

  • 95% of our c++ code is being built perfectly with scons -j4. Dependcies tree is properly created
  • Our custom python builder generates the .jar from java component with ant

What is NOT working :

  • Our custom python builder to generate binaries from grammar files component is being called BEFORE the java compilation when using -j4.

The following is the scons script file that generate sideLibrary.jar (using Command)

Import('common_env')
#Grab a copy of the top environment (the one sent by the SConstruct file)
common_env = common_env.Clone()
#Current working directory.
currentDir = common_env['ENV']['LOCALROOT'] + '/' + common_env.sconsGetCWD()
#Launches ant -q
tgts = common_env.Command(target = [currentDir + '/dist', currentDir + '/dialogeditor/lib', currentDir + '/doc/sdtk', currentDir + '/obj', common_env['ENV']['LOCALROOT'] + '/LINK/bin/win32/sideLibrary.jar'], source = 'build.xml', action = 'ant -q -f $SOURCE')
#Cleans the generated folders
common_env.Clean(tgts, '')

The following is the scons script file to generate the grammar binary files (using custom builder) :

Import('common_env')
#Creates a copy of the build environment sent from the SConstruct file.
common_env = common_env.Clone()
#Because this component is only compiled in win32
if (common_env['ENV']['CONFIG'] == "win32"):
    #Language of the grammar.
    common_env['ENV']['LANGUAGE'] = 'en'
    #Language pack.
    common_env['ENV']['LANGUAGE_PACK'] = 'English.America.3.3.0'
    #Grammar name
    common_env['ENV']['GRAMMAR_NAME'] = 'faa'
    #Compiles the grammar using a custom builder.
    common_env.buildGrammar(src = common_env['ENV']['LOCALROOT'] + '/LINK/bin/win32/sideLibrary.jar')

Someone told me that passing src = common_env['ENV']['LOCALROOT'] + '/LINK/bin/win32/sideLibrary.jar' to our custom builder tells scons that he MUST automagically wait fro sideLibrary.jar before calling buildGrammar. I really have doubts because the doc only says :

source: A list of Node objects representing the sources to be used by this builder function to build the targets.

That brings me to Scanner, ParseDepens and Depends (I already read scons documentation about them).

It is not really clear in my mind which one I should use to accomplish what I want to do : force compilation of java component BEFORE another component with -j4.

Could someone can light my lantern please?

Thank you

EDIT (added dependency tree)

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
copyAction(["LINK\bin\win32\msvcp100d.dll", "LINK\bin\win32\msvcr100d.dll", "LINK\bin\win32\mfc100d.dll", "LINK\bin\win32\mfc100ud.dll", "LINK\bin\win32\rdbgsetup_x86.exe"], ["C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcp100d.dll", "C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcr100d.dll", "C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100d.dll", "C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100ud.dll", "C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\RemoteDebugger\rdbgsetup_x86.exe"])
ant -q -f sr\side\build.xml
buildGrammar(["LINK\data\Grammars\src\grammars\AvailableCommand.xml", "sr\sra\src\grammars\en\faa\faa.autopron", "sr\sra\src\grammars\en\faa\faa.missing", "sr\sra\src\grammars\en\faa\faa.functions"], [])
+-.
  +-build
  | +-build\debug
  |   +-build\debug\sr
  |     +-build\debug\sr\side
  |     | +-build\debug\sr\side\win32
  |     |   +-sr\side\build.xml
  |     |   +-sr\side\side.sc
  |     +-build\debug\sr\sra
  |       +-build\debug\sr\sra\src
  |         +-build\debug\sr\sra\src\grammars
  |           +-build\debug\sr\sra\src\grammars\win32
  |             +-sr\sra\src\grammars\src_grammars.sc
  +-c
  +-cm
  | +-cm\sconsUtilities
  +-LINK
  | +-LINK\bin
  | | +-LINK\bin\win32
  | |   +-LINK\bin\win32\mfc100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcp100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcr100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100ud.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\RemoteDebugger\rdbgsetup_x86.exe
  | |   +-LINK\bin\win32\mfc100ud.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcp100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcr100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100ud.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\RemoteDebugger\rdbgsetup_x86.exe
  | |   +-LINK\bin\win32\msvcp100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcp100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcr100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100ud.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\RemoteDebugger\rdbgsetup_x86.exe
  | |   +-LINK\bin\win32\msvcr100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcp100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcr100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100ud.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\RemoteDebugger\rdbgsetup_x86.exe
  | |   +-LINK\bin\win32\rdbgsetup_x86.exe
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcp100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugCRT\msvcr100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100d.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC100.DebugMFC\mfc100ud.dll
  | |   | +-C:\svn\3rdParty\devTool\Microsoft_Visual_Studio_10.0\RemoteDebugger\rdbgsetup_x86.exe
  | |   +-LINK\bin\win32\sideLibrary.jar
  | |     +-sr\side\build.xml
  | |     +-C:\svn\3rdParty\devTool\apache-ant-1.9.2\bin\ant.BAT
  | +-LINK\data
  |   +-LINK\data\Grammars
  |     +-LINK\data\Grammars\src
  |       +-LINK\data\Grammars\src\grammars
  |         +-LINK\data\Grammars\src\grammars\AvailableCommand.xml
  +-SConstruct
  +-sr
    +-sr\side
    | +-sr\side\build.xml
    | +-sr\side\dialogeditor
    | | +-sr\side\dialogeditor\lib
    | |   +-sr\side\build.xml
    | |   +-C:\svn\3rdParty\devTool\apache-ant-1.9.2\bin\ant.BAT
    | +-sr\side\dist
    | | +-sr\side\build.xml
    | | +-C:\svn\3rdParty\devTool\apache-ant-1.9.2\bin\ant.BAT
    | +-sr\side\doc
    | | +-sr\side\doc\sdtk
    | |   +-sr\side\build.xml
    | |   +-C:\svn\3rdParty\devTool\apache-ant-1.9.2\bin\ant.BAT
    | +-sr\side\obj
    | | +-sr\side\build.xml
    | | +-C:\svn\3rdParty\devTool\apache-ant-1.9.2\bin\ant.BAT
    | +-sr\side\side.sc
    +-sr\sra
      +-sr\sra\src
        +-sr\sra\src\grammars
          +-sr\sra\src\grammars\en
          | +-sr\sra\src\grammars\en\faa
          |   +-sr\sra\src\grammars\en\faa\faa.autopron
          |   +-sr\sra\src\grammars\en\faa\faa.functions
          |   +-sr\sra\src\grammars\en\faa\faa.missing
          +-sr\sra\src\grammars\src_grammars.sc
scons: done building targets.

EDIT 2 added buildGrammar

def buildGrammar(target, source, env):
    LANGUAGE = env['ENV']['LANGUAGE']
    LANGUAGE_PACK = env['ENV']['LANGUAGE_PACK']
    GRAMMAR_NAME = env['ENV']['GRAMMAR_NAME']
    LINKD = env['ENV']['LINKD']
    SIDE_BINARY_DIR = LINKD + '/bin/win32'
    DATAPACK_OUT_UDIR = DATAPACK_OUT_DIR = LINKD
    GRAMMAR_SRC_DIR = env['ENV']['LOCALROOT'] + '/sr/sra/src/grammars/' + LANGUAGE + '/' + GRAMMAR_NAME
    GRAMMAR_SRC_FILES = glob.glob(GRAMMAR_SRC_DIR + '/*.grammar') + glob.glob(GRAMMAR_SRC_DIR + '/*.dictionary')
    CUSTOM_NL_FUNCTIONS_FILE = GRAMMAR_SRC_DIR + '/' + GRAMMAR_NAME + '.functions'
    PBS_FILE = env['ENV']['LOCALROOT'] + '/fwk/systemInterface/simnet/src/pbs_api.h'
    SIDE_LIBRARY_PATH = SIDE_BINARY_DIR + '/sideLibrary.jar'
    SRROOT = env['ENV']['LOCALROOT'] + '/sr'
    GRM_PROJECT = 'faa'
    UTIL_DIR = env['ENV']['LOCALROOT'] + '/sr/sra/utils/build'
    #CUSTOM_NL_FUNCTIONS_FILE
    print "(+) Generating " + CUSTOM_NL_FUNCTIONS_FILE
    customNlFunctionsFile = open(CUSTOM_NL_FUNCTIONS_FILE, "w")
    call("perl " + UTIL_DIR + '/header2Nuance ' + PBS_FILE, stdout = customNlFunctionsFile) 
    customNlFunctionsFile.close()
    print "Done."
    #dataPackBuild
    print "(+) Building lexix datapack... (no find and no chmod) - Grammar"
    if os.path.exists(DATAPACK_OUT_UDIR + '/data/Grammars/abnf/bin/' + GRAMMAR_NAME):
        shutil.rmtree(DATAPACK_OUT_UDIR + '/data/Grammars/abnf/bin/' + GRAMMAR_NAME)
    if os.path.exists(DATAPACK_OUT_UDIR + '/data/SOURCES'):
        shutil.rmtree(DATAPACK_OUT_UDIR + '/data/SOURCES')
    linkToWD = env['ENV']['LOCALROOT'] + '/sr/sra/src/grammars'
    os.chdir(linkToWD)
    call('java -jar ' + SIDE_LIBRARY_PATH + ' --verbyx -l ' + SIDE_BINARY_DIR + '/plugin.properties ' + '--deploy . ' + DATAPACK_OUT_DIR + ' > /dev/nul')
    if not os.path.exists(DATAPACK_OUT_UDIR + '/data/Grammars/src/grammars/en/faa'):
        os.makedirs(DATAPACK_OUT_UDIR + '/data/Grammars/src/grammars/en/faa')
    for file in GRAMMAR_SRC_FILES:
        shutil.copy(file, DATAPACK_OUT_UDIR + '/data/Grammars/src/grammars/en/faa')
    for file in glob.glob('*.xml'):
        shutil.copy(file, DATAPACK_OUT_UDIR + '/data/Grammars/src/grammars')
    shutil.copy('.project', DATAPACK_OUT_UDIR + '/data/Grammars/src/grammars')
    copytree('./project', DATAPACK_OUT_UDIR + '/data/Grammars/src/grammars/project')
    copytree('./.settings', DATAPACK_OUT_UDIR + '/data/Grammars/src/grammars/.settings')
    os.chdir(env['ENV']['LOCALROOT'])
    print "Done."

    #grammar.mk
    DATA_DIR = env['ENV']['LOCALROOT'] + '/LINK/data'
    GRAMMAR_SRC_DIR = SRROOT + '/sra/src/grammars'
    GRAMMAR_PROJECT_SRC_DIR = GRAMMAR_SRC_DIR + '/en/' + GRM_PROJECT
    GRAMMAR_OBJ_DIR = DATA_DIR + '/Grammars/bin'
    OUTPUT_DIR = GRAMMAR_OBJ_DIR + '/' + LANGUAGE + '/' + GRM_PROJECT
    AD_HOME = 'C:/Adacel/temp'
    PROJECT_DICTIONARY = DATA_DIR + '/Grammars/bin/dictionary'
    PROJECT_SRC_DICTIONARY = GRM_PROJECT + '.dictionary'
    PROJECT_OUT_DICTIONARY = OVERRIDE_OUT_DICTIONARY = PROJECT_DICTIONARY + '/' + LANGUAGE_PACK
    PROJECT_OUT_DIST_DICTIONARY = OVERRIDE_OUT_DIST_DICTIONARY = AD_HOME + '/nuance/dictionary/' + LANGUAGE_PACK
    PROJECT_SRC_OVR_DICTIONARY = OVR_DICTIONARY = 'override.dictionary'
    GRAMMAR_MASTER_FILE = GRM_PROJECT + '.grammar'
    COMPILE_FLAGS = LANGUAGE_PACK + ' -auto_pron -optimize_graph -dont_flatten -node_array_optimization_level FULL -override_dictionary ' + OVERRIDE_OUT_DIST_DICTIONARY + '/' + OVR_DICTIONARY + ' -merge_dictionary ' + PROJECT_OUT_DIST_DICTIONARY + '/' + PROJECT_SRC_DICTIONARY
    print "(+) Merging " + PROJECT_OUT_DICTIONARY + " and " + PROJECT_OUT_DIST_DICTIONARY
    if not os.path.exists(PROJECT_OUT_DICTIONARY):
        os.makedirs(PROJECT_OUT_DICTIONARY)
    shutil.copy(SRROOT + '/sra/src/grammars/' + LANGUAGE + '/' + GRM_PROJECT + '/' + PROJECT_SRC_DICTIONARY, PROJECT_OUT_DICTIONARY)
    if not os.path.exists(PROJECT_OUT_DIST_DICTIONARY):
        os.makedirs(PROJECT_OUT_DIST_DICTIONARY)
    shutil.copy(SRROOT + '/sra/src/grammars/' + LANGUAGE + '/' + GRM_PROJECT + '/' + PROJECT_SRC_DICTIONARY, PROJECT_OUT_DIST_DICTIONARY)
    print "Done."
    print "Copying " + OVERRIDE_OUT_DICTIONARY + " and " + OVERRIDE_OUT_DIST_DICTIONARY
    if not os.path.exists(OVERRIDE_OUT_DICTIONARY):
        os.makedirs(OVERRIDE_OUT_DICTIONARY)
    shutil.copy(SRROOT + '/sra/src/grammars/' + LANGUAGE + '/' + GRM_PROJECT + '/' + PROJECT_SRC_OVR_DICTIONARY, PROJECT_OUT_DICTIONARY)
    if not os.path.exists(OVERRIDE_OUT_DIST_DICTIONARY):
        os.makedirs(OVERRIDE_OUT_DIST_DICTIONARY)
    shutil.copy(SRROOT + '/sra/src/grammars/' + LANGUAGE + '/' + GRM_PROJECT + '/' + PROJECT_SRC_OVR_DICTIONARY, OVERRIDE_OUT_DIST_DICTIONARY)
    print "Done."
    NUANCE_COMPILER = env['ENV']['NUANCE_ROOT'] + '/bin/win32/nuance-compile '
    #Compiles Grammar.
    if not os.path.exists(OUTPUT_DIR + '/' + LANGUAGE_PACK):
        os.makedirs(OUTPUT_DIR + '/' + LANGUAGE_PACK)
    print "(+) Building " + LANGUAGE + "." + GRM_PROJECT + "." + LANGUAGE_PACK + " grammar..."
    os.chdir(env['ENV']['LOCALROOT'] + '/sr/sra/src/grammars/' + LANGUAGE + '/' + GRM_PROJECT)
    compilationCmd = NUANCE_COMPILER + GRAMMAR_MASTER_FILE + ' ' + COMPILE_FLAGS + ' -o ' + OUTPUT_DIR + '/' + LANGUAGE_PACK + ' -write_auto_pron_output ' + OUTPUT_DIR + '/' + LANGUAGE_PACK + '/.missing'
    call(compilationCmd)
    os.chdir(env['ENV']['LOCALROOT'])
    print "Done."

    #Grammar-database.mk
    print "(+) Creating dynamic grammar database (" + LANGUAGE + ")..."
    GRAMMAR_DIR = env['ENV']['LOCALROOT'] + '/LINK/data/Grammars/bin'
    GRAMMAR_DATABASE_DIR = GRAMMAR_DIR + '/' + LANGUAGE + '/dgdb-common'
    if os.path.exists(GRAMMAR_DATABASE_DIR):
        shutil.rmtree(GRAMMAR_DATABASE_DIR)
    if not os.path.exists(GRAMMAR_DIR + '/' + LANGUAGE):
        os.makedirs(GRAMMAR_DIR + '/' + LANGUAGE)
    NUANCE_DATABASE = env['ENV']['NUANCE_ROOT'] + '/bin/win32/nuance-database-admin '
    DB_FLAGS = '-dboperation create -dbname common -dbprovider fs -dbclass dgdb -dbroot ' + GRAMMAR_DIR + '/' + LANGUAGE
    call(NUANCE_DATABASE + DB_FLAGS)
    print "Done."
    return None
Solution

  • Well, I am not sure this question will ever help someone, but I feel important to at least answer my question.

    So, to everyone starting with scons, always use "source=" as function parameter. All my problem (-j4 not working), was completely related with the fact that when I was calling my custom builder, I was using it like that "buildGrammar(src="blabla")". It is wrong. Must be "buildGrammar(source="blabla")