Fix a crash caused by incorrect assumptions Reported here https://github.com/llvm/llvm-project/pull/72644#discussion_r1469525524