A Zhejiang University and Swansea University team took an unintuitive finding to IJCAI: when fine-tuning a vision-language model on new data, tuning the text tower helps and tuning the image tower can hurt. Their Adaptive Asymmetric Adapter automates that choice, applying the brakes to visual tuning exactly when it would damage the model.
Three rules from eleven datasets
Across 11 vision datasets the team found that text-encoder tuning outperforms image-encoder tuning on most tasks, because text carries abstract semantics cheaply. On in-distribution tasks, tuning both wins. But on out-of-distribution tasks, aggressive image tuning wrecked CLIP’s generalisation: the model “went near-sighted” on data it had never seen.
The fix is a confidence gate. The adapter scores how sure the model is; when confidence is high it lets the image adapter fit, and when confidence is low it forces the image correction toward zero, reverting to the pretrained visual features. No manual switch, no human watching the stream.
Proof on the board
Against 11 strong baselines the method led on base-to-novel, cross-dataset and domain-generalisation tests. Ablations confirmed the brake is load-bearing: remove it and novel-class performance drops; add a brake to the text side and performance falls, since text should stay open. The lesson is restraint: keep the pretrained base, and let go only when the data says so.
Editor’s note: This is an adapted translation of the original LeiFeng Network report. It has been trimmed and restructured for readability for an international business audience.