Why does MIXER_B/MIXER_F introduce an additional 6 dB or more gain on top of the conversion gain when I measure total channel power?
There are two primary reasons for this. By default, when upconverting from baseband MIXER_B and MIXER_F operate as two separate mixers, as in an I/Q modulator. One mixer upconverts the I channel, the other mixer upconverts the Q channel using the LO phase shifted by 90 degrees. The outputs of the two channel mixers are then combined to form the output of the main mixer. Because each channel mixer has the same settings as the main mixer, a 3 dB gain on top of the gain conversion results because two signals are being added together.
Another 3 dB gain is due to the mixer generating both sidebands. Either filter the output of the mixer for only one sideband, or, if measuring the power directly, specify the frequency band of only one sideband in the CH_PWR measurement. The LO feedthrough due to low LO to RF/IF isolation is another source of excess measured power.
The default LO to RF/IF isolation is 25 dB, which is set through the LO2OUT parameter. When upconverting from baseband, the LO feedthrough falls in the middle of the upconverted spectrum.