It looks like the SDO might have been the culprit.
Which is weird - my other BMP280 had no issues except that I connected it on a breadboard so maybe somehow to metal contact on the breadboard helps? Or manufacturing tolerance?
Anyway, I ground SDO, and am so far having no abnormal readings so far.