Search code examples
unreal-engine4

How do you run tests from the command line?

To do this in-editor you open the automation tab, connect to the session and choose which tests to run.

How do you do it from the command line?

(NB. not compiling UnrealEngine/Engine/Build/BatchFiles/* comprehensively covers both building the application and compiling it. Specifically, given that you have code that is 100% happy to compile, how do you kick the test suite off)

--

Here's some more info, from recent testing on 4.10:

Running tests from the editor:

UE4Editor Project.uproject -ExecCmds="Automation RunTests MyTest"

Notice the absence of the -Game flag; this launches the Editor and runs the tests successfully in the editor console.

Running the game engine and using the 'popup log window':

UE4Editor Project.uproject -Game -ExecCmds="Automation RunTests MyTest" -log

This runs the game in 'play' mode, pops up an editor window; however, the logs stop at:

LogAssetRegistry: FAssetRegistry took 0.0004 seconds to start up

...and the game never closes or executes the tests.

Running the game engine and logging to a file:

UE4Editor Project.uproject -Game -ExecCmds="Automation RunTests MyTest" -log=Log.txt

This runs the game in 'play' mode, and then stops and never exists.

It does not appear to run any tests or log to any files.

The folder Saved/Logs does not exist after quitting the running game.

Running in the editor, test types, etc...

see: https://answers.unrealengine.com/questions/358821/hot-reload-does-not-re-compile-automation-tests.html,

Hot reload is not supported for tests; so this isn't an option.

There's also been some suggestion in various places that the test type (eg. ATF_Game, ATF_Editor) has some affect on if runs are or can be run; perhaps this is an issue to, but I've tried all kind of combinations with no success.

--

I've tried all kinds of combinations of things trying to get this working, with no success so it's time for a bounty.

I'll accept an answer which reliably:

  • Executes a specific test from the command line
  • Logs the output from that test to a file
Solution

  • Right, no one has any idea here or on the issue tracker.

    After some serious digging through the UE4 source code, here's the actual deal, which I leave here for the next suffering soul who can't figure this out:

    To run tests from the command line, and log the output and exit after the test run use:

    UE4Editor.exe path/to/project/TestProject.uproject
              -ExecCmds="Automation RunTests SourceTests"
              -unattended
              -nopause
              -testexit="Automation Test Queue Empty"
              -log=output.txt
              -game

    On OSX use UE4Editor.app/Contents/MacOS/UE4Editor.

    Notice that the logs will, regardless of what you supply, ultimately be placed in:

    WindowsNoEditor/TestProject/Saved/Logs/output.txt

    or

    ~/Library/Logs/TestProject/output.txt

    Notice that for mac this is outside of your project directory, in, for example, /Users/doug/Library/Logs/TestProject. (Who thought that was a good idea?)

    (see https://wiki.unrealengine.com/Locating_Project_Logs#Game_Logs)

    You can list automation tests using:

    -ExecCmds="Automation List"

    ...and then parse the response to find tests to run; automation commands may be chained, for example:

    -ExecCmds="Automation List, Automation RunAll"