The term "application instance" is used in all public documentation, but in the LabVIEW C++ code, those same things are called "contexts". Because R&D folks occasionally post to customers without tech writers reading over our shoulders, we slip up and use the other term. So you should generally regard those terms as interchangeable, but know that "application instance" is preferred for a long list of (very good) reasons.
An app instance is very nearly an isolated LabVIEW.exe. The VIs in one app instance cannot interact with the VIs in another context unless they do the same things that would be required to talk to a VI that was loaded on a separate machine entirely. So whatever you would do to talk to a network VI is the stuff you generally have to do to talk to a VI in another app instance.
Application Refnums are the references to an app instance. Prior to 8.0, you would open an app reference to an entire LabVIEW execution system. Now you open an app reference to one app instance within the execution system.
All LabVIEW files -- VIs, libraries, projects -- load into an app instance. The same file may be loaded into multiple app instances simultaneously. Editing one of the "reflections" (a term that R&D uses which is distinct from "clones" of a VI, which has to do with reentrancy) does not change the other reflections of that file until you synchronize the app instances (either by pressing that green arrow button that appears on the VI next to the Run arrow OR by saving the file).
Projects are loaded into an app instance, but they also manage a set of app instances. The "My Computer" instance is the one that the project is actually loaded into. If you add any targets to a project -- an FPGA target, RT target, etc -- those are other app instances. *All* of the app instances that a project creates, including the "My Computer" instance, are life-time managed by the project, meaning that when you close the project, you close the app instance, which closes out all the VIs and Libraries loaded into those app instances.
Any given app instance may be owned by zero or one projects, but not more.
LabVIEW has multiple private app instances, including the "NI.LV.Dialog" app instance which is where VIs from the Tools menu run. These private contexts allow tools written with VIs to run without interfering with the user app instances, so there is never a chance of one of those VIs cross linking with user VIs or keeping a user VI from being able to load because a VI of that name is already in memory. Various parts of LabVIEW will create temporary app instances -- for example, AppBuilder creates a temporary app instance which it loads all the source VIs into so that it can rename them, strip diagrams, etc, without touching the user's original source files.
Users can create their own temporary app instances. I believe the Open Application Refnum primitive can be used for this by providing a unique port number to the primitive. There are other methods as well. This is an area of LabVIEW where my knowledge is pretty weak and others may be able to fill in more details.