The windows installer database actually has a bunch of transactional facilities built into it that can make this pretty nicely handled automatically.
What you're going to want to do however, is execute the msiexec.exe not as a child process, and then exit your process while it's running. Let the installer have a last step of "launch the app" so when it's finished it'll reopen the new application, and it'll be pretty seamless then.
If your app is being updated and open and a reboot happens, the transactional rollbacks may have minor issue with files being in a funny state because they were in use when the MSI attempted to do it's update. If your MSI is well constructed though, while it's being installed, a reboot (even unprompted) will result in either a full rollback or a transaction continuation beginning when the restart completes.
I suspect you do not have your msi setup to execute the file updates in the deferred mode correctly
Per wikipedia (emphasis mine)
[...] Deferred mode. In this phase, the script built in immediate mode is executed in the context of the privileged Windows Installer service. The script must be executed by a privileged account because of the heterogeneity of the scenarios in which a setup operation is initiated. For example, elevated privileges are necessary to serve on-demand installation requests from non-privileged users. (To run with elevated privileges, however, the package must be deployed by a local administrator or advertised by a system administrator using Group Policy.)
Rollback
All installation operations are transactional.[7] In other words, for each operation that Windows Installer performs, it generates an equivalent undo operation that would revert the change made to the system. In case any script action fails during deferred execution, or the operation is cancelled by the user, all the actions performed until that point are rolled back, restoring the system to its original state. Standard Windows Installer actions automatically write information into a rollback script; package authors who create custom actions that change the target system should also create corresponding rollback actions (as well as uninstall actions and uninstallation-rollback actions). As a design feature, if applied correctly this mechanism will also roll back a failed uninstall of an application to a good working state. [...]