Generally stuff like that is held against someone when they didn’t even know their dad died, or didn’t realize that that particular person would overreact by being reminded of something that doesn’t seem associated.

Basically, caring far more about someone’s reaction than intent (or lack thereof) that accidentally upsetting someone is breaking a social norm.