I'm a beginner programmer ( I know scripting and basic C++). I'm using UnrealEngine5/C++ and want to update some variables inside engine using my own programs (Lisp) at runtime. Currently I'm using text file as a buffer. Are there better ways? I don't want each part of engine have a plug for constantly checking this file for updated values. I want general loop of engine remain intact. I don't know if you should allocate same memory addresses for variables in different programs as a solution - would like to know about established ways of doing it before inventing the wheel.
I understand that answer can be complex but if you can at least guide me in terms of books and concepts that I need to understand in order to make such setup that I won't need a buffer file.
There are quite a few ways to do what you want, and they are mostly not specific to Lisp. So, I'd like to give you some general advice and add a few Lisp-related comments where appropriate.
Basically, there are two extremes among the different approaches you might decide to take:
Tight coupling is usually preferred when you want to remain primarily in the realm of your base language, and when maximum efficiency of the program execution is the primary goal. With tight coupling, you may use such tools as shared libraries and foreign-function interface, i.e. you'll embed the scripting language into your base runtime. With C/C++ as a base and Lisp as an embedded language, you may take a look at ECL and CLASP implementations which target this use case specifically.
Tight coupling, however, is usually somewhat limiting to the embedded language, it gives you less flexibility. If your priority and main development effort will be in it (or, at least, in both platforms more or less on par), loose coupling may be more attractive. With it, you will run two (or more) separate processes and use IPC to share data and/or commands between them. There are many approaches to IPC. Using a file to share data is one of them, albeit one of the most primitive ones. A more advanced approach that might be convenient for your use case is using Unix domain sockets (as the most efficient) or the other types of sockets (plain network sockets or even websockets). Regardless of a particular mechanism selected, the basic principle is that your data should be marshaled between two separate environments, which introduces overhead. Yet, it also may help you structure the program much better. As for marshaling, once again there's a variety of approaches that also range from a more-coupled one (represented by using platform-specific binary encodings, such as Python pickle format) to less-coupled (using standard text-based data-interchange formats such as JSON).
I'd say that, in general, for interfacing between such conceptually different platforms as C and Lisp, the loose coupling approach is the default as it doesn't force the paradigms and conventions of each of these environments onto each other. However, if what you have to achieve is a limited one-off task, it might be much less effort to integrate Lisp into C.