A reentrant VI is a VI that has a separate data space allocated for each instance of the VI. Multiple instances of reentrant VIs can execute in parallel without interfering with each other. Non-reentrant VIs have a data space that is shared between multiple calls to the VI.
First, consider three VIs, foo.vi, bar.vi, and sub.vi. Both foo.vi and bar.vi call sub.vi.
If sub.vi is normal (not reentrant), then if foo.vi tries to call sub.vi but sub.vi is busy servicing a call from bar.vi, then foo.vi has to wait. This can be both a very GOOD thing and a very BAD thing depending on circumstances. It's very GOOD when sub.vi controls access to something like a serial port, where you only want one part of your program accessing it at a time. It's very BAD when sub.vi controls something like ALL the serial ports, because you may want foo.vi to be able to use one serial port while bar.vi is busy using a different serial port. Another very bad circumstance is where foo.vi is in a critical loop and bar.vi is not, yet because of the contention for sub.vi, bar.vi can end up blocking foo.vi.
If sub.vi is reentrant, then both foo.vi and bar.vi can call sub.vi at the same time. In order for this to work, each call to sub.vi needs to have its own "data space" which is all the internal storage sub.vi uses in order to execute its code.
Now at this point I need to point out a distinction between LabVIEW and most other languages. LabVIEW doesn't want to allocate a data space on the fly, because for LabVIEW that would slow down performance. LabVIEW allocates all the VI data spaces it needs when VIs are being loaded. Except when you use VI Server to call VIs dynamically, all the loading happens before any VIs execute. Therefore, for each place that sub.vi appears on foo.vi's block diagram, a copy of sub.vi's data space gets embedded in foo.vi's data space, assuming sub.vi is reentrant. If sub.vi isn't reentrant, it just has its one data space allocated that each call will use in turn.
In most (all?) other languages, reentrant functions allocate their data spaces on the fly, so there's no storage that goes with each place that a particular function is called.
How does LabVIEW's unusual implementation affect us in practical terms? There are really two ways:
1. If you use uninitialized shift registers to store information, then you can get two different behaviors depending on the reentrancy of your VI. For a non-reentrant VI, you get a data sharing function that lets you move large quantities of information between parallel loops without making copies of it. For a reentrant VI, you get a reusable storage function that can keep independent copies each place you use it.
2. The second implication is that you can't do recursion (functions that call themselves) easily in LabVIEW. In most languages, if a function is reentrant then it's OK for it to call itself. In LabVIEW, that would require that the data storage for sub.vi would include a copy of the data storage for sub.vi which would ... to infinity. You can do recursion in LabVIEW if you use VI Server to have a VI call itself dynamically, but as I said, allocating data spaces on the fly is inherently slow. In LabVIEW, it's best to convert recursive algorithms to their iterative equivalents, which I hear is mathematically proven to always be possible. In the iterative version, you'll end up changing the sizes of arrays at each iteration, which is also one of the slower operations in LabVIEW, but is not nearly as slow as dynamic VI calls.
To expand on this, reentrant means that more than one execution is allowed to take place at the same time. In other languages, it is more a situation than a setting. You never mark a C function as allowing or disallowing reentrancy, it is either safe to do so or a source of bugs. In LV it is a setting, and many times its setting doesn't affect the correctness of a VI, but in some cases, it can be a source of bugs. It depends on what the VI does.
The setting in LV determines two major attributes about how a VI executes.
First is access. With reentrancy turned off, only one call to the subVI can be active at a time. When the current call finishes, the next one can begin. The subVI calls queue up while the VI is busy. For functions that execute quickly, this is normally fine and reentrancy doesn't affect much.
If you have a function that uses TCP to talk to another computer and waits for responses, these waits also affect the other subVI calls that are queued up. So if you have an operation that can occur in parallel and doesn't consume the CPU, you can make the VI reentrant and the multiple subVI calls don't enter a queue, and multiple VIs can talk TCP and wait for responses at once. This allows the wait time of one subVI to be used as work time in another and increases overall performance.
On the otherhand, given a VI that reads a global modifies it and writes it back, a reentrant subVI means that more than one subVI call at a time can be modifying the global -- a race condition which will cause incorrect answers. Lots of real-world devices also get confused when more than one subVI tries to control them at a time. So when trying to protect a global resource, the one of the tools, and frequently the easiest to use is to simply make sure that the access goes through a non-reentrant VI.
The second attribute is data side-effects. If a VI has unconnected controls or uninitialized shift registers on its diagram, then it remembers some amount of information from call to call. A good example of this is a PID or a filter. Data from previous calls affect the result of the next call. For these sorts of VIs, if they are reentrant, then each call gets its own place to store the previous call's state information. If made non-reentrant, there will be only one storage location for all calls to share, so the data will get all jumbled, likely causing an incorrect answer.