I am developing a Chess Game in Unity. I am not writing the AI by myself but using the chess engine named MadChess.
I downloaded the source code of the engine(which is written in C#) as well as the executable file named "MadChess.exe"(which is basically a command prompt through which we can talk with the engine following UCI protocols)
I haven't actually integrated a chess engine with a unity project before and also not able to find many resources online on how to do it.
Approach 1: Initially, I wrote a C# script in Unity which creates a separate process to communicate with "MadChess.exe" but I later realized that the approach is buggy. This is because when I build my Unity Project as a standalone application for Windows, it is dependent on the physical "MadChess.exe" file located on my PC i.e. it doesn't build everything(My Unity Project code along with the Chess Engine Code) together into a single executable application. This according to me is faulty and can only be rectified by creating an installer of the game which injects the "MadChess.exe" file along with my main Unity code.
Approach 2: Now, I am trying to go through the source code of MadChess chess engine project and basically creating .dll files of the chess engine project which I plan to use as plugins in my Unity project. I would interface with them through my C# code in Unity eventually. Is it the correct way or not?
I am stuck at this for around a week or so and still not able to find a clear way. I don't want to spend another week just to realize that I am doing it wrong. Even if I am wrong from the start please let me know. I simply want to integrate a Chess Engine with my Unity Project. If someone has any suggestions or ideas or if someone has done this before please guide me. I would be grateful. Thanks for bearing with me till here.
I recommend using Approach 1
Both approaches sound valid, but I recommend using Approach 1 to communicate with MadChess.exe as a process, because this is much less work than Approach 2, especially if you already have the first approach working (aside from the build issue).
Also, aside from the source code, if the MadChess project is only providing the exe and not the dll files, then I think it's safer to stick with the approach that is implicitly "recommended" by the project. You don't know what kinds of issues you'll run into when you try to make the dll files yourself (and it sounds like you have been running into issues already).
Even though it should be possible to use dll files, I think it's preferable to pick the approach that will let you progress on your project faster. You can always switch to Approach 2 later if you run into other issues.
(Just make sure to show appropriate errors in case MadChess.exe is not found, so that you can troubleshoot issues quickly.)
How to include MadChess.exe as part of your build
You don't necessarily need an installer to include the MadChess.exe file with your game. You can also copy MadChess.exe as part of the Build Player Pipeline. Using either BuildPipeline.BuildPlayer or the PostProcessBuildAttribute, you can get the path of the built project, and then use FileUtil.CopyFileOrDirectory to copy MadChess.exe during the build step.
Also, as @AdrianSgro mentioned in a comment, StreamingAssets works as well. (But note that the documentation explicitly mentions that .dll files in this folder are not included if you need to include any dll dependencies.)