robot_localization: Differential parameters and covariance

I'm trying to fuse visual odometry with an IMU sensor to get pose estimates. Neither of these give covariance by default, so I am setting them myself. My question is: Is at least one absolute position required for the covariance to converge? Is this expected behaviour or is something wrong? For whatever values I try, unless I use the visual odometry with differential=False, the covariance increases forever. I have tried having very small covariance diagonals in the messages, slightly bigger in the noise, and large (but not huge) for the initial estimate.