Its interesting that you see a difference there, but it may be because of the slightly higher loss that harsh is talking about. I assume that the modulator you are using in such an instance is using a channel far from any broadcast channel you're attempting to receive. Maybe a channel 3/4 modulator unless you're in one of the few areas with a low VHF station, or maybe a high UHF channel since channel 51 is the highest currently (and the FCC is trying to move them off 51 because of interference in the lower LTE bands)
If your modulator was an ideal transmitter it would generate only the frequency it is supposed to, and nothing else. Given that even channel 51 transmitters at TV stations aren't ideal transmitters (hence the interference with LTE) then obviously yours isn't either. Unless it is very high end, it probably puts out a lot of stray RF across a lot of frequencies - certainly at least the harmonics of the desired frequency. Normally this wouldn't matter, but when you mix the signal with OTA where any added noise can kill fringe reception it could be a problem. In such a case you're better off if the splitter has a bit more loss, as it'll attenuate those stray frequencies better. If you're working with fringe OTA reception, even a single db of additional noise could make or break you.
Port to port isolation shouldn't be too much a factor in the performance of a splitter when used as a simple combiner. If you leak signal from the OTA leg it'll go into the modulator, which doesn't matter. If you leak signal from the modulator it'll be broadcast from your antenna (only a problem if it is really bad and the FCC shows up at your doorstep ) Now obviously any signal that leaks from the OTA leg is signal that isn't making it into the combined signal, but the difference between a port to port isolation of 20db and 40db leaves much less than 1 db of signal that's lost. I would guess you're better off with a splitter that has high insertion loss than one that has high port to port isolation. That assumes that a splitter with high insertion loss when used in the normal direction is also losing more signal when used as a combiner. That assumption might be wrong.
One thing you could try to test if I'm right is to attenuate the signal coming from your modulator before combining it with the OTA signal. Most modulators output pretty powerful signals, so the noise will be powerful as well. If you damp that signal down to a lower but still usable level you'd also damp down the noise. You could do with that an attenuator or a splitter or two inline. However much you can drop the signal and not hurt the quality of the picture from the modulator. Give that a try sometime, I'd be interested to hear if it helps as much as using a 2.4 GHz splitter when combining signals.
For me, the big difference is that I'm not using splitters as a combiner (well, am I in one spot, but that's not where I noticed the type of splitter having any visible effect on my results) Instead I'm using a splitter as both a splitter and a combiner simultaneously. The signal coming out of the modulator into the "in" leg of a splitter caused a double image on the screen when enough of it leaked out the other leg of the splitter, which is connected to the TV's tuner.
Likewise, at my "head end" I have another splitter that's both a splitter and combiner simultaneously. A splitter for the modulator signal coming from the coax that originates at that TV (i.e. from that other splitter/combiner) with one leg connected to another splitter that acts as a combiner only to mix the modulated signal with the CATV signal which then goes into the 25db amp. The other leg of that splitter/combiner is connected to one of the outputs of a 16 way splitter. Obviously that leg is intended as an input, to feed the CATV signal to the TV on the other end of that coax, but I do have half my modulated signal traveling out that leg into the 16 way splitter.
I don't know what the port to port isolation numbers on such a big splitter are, but I doubt it is too much of a problem. Most of the signal would travel back through that splitter's input and hit the output of another two way splitter that serves the two 16 ways. Any signal that makes it through there will stop cold when it hits the amplifier's output, so I'm not too concerned about interference from that half of the modulator's signal that's "lost".