Close https://github.com/llvm/llvm-project/issues/61064.
The root cause of the issue is that we will deserilize some declarations
eagerly when reading the BMI. However, many declarations in the BMI are
not necessary for the importer. So it wastes a lot of time.
