Awkward moments in foster care: Secret pregnancy

Our church is large enough that we have many acquaintances or old college friends that we only bump into occasionally.  Not all of them know we're foster parents.  So, one guy saw me holding a foster baby that has our same skin color and I could see his face immediately flood with confusion.  "Oh hey!  Wow, I didn't know... I mean, I must not be paying attention.  I don't even remember you being pregnant..."  More awkward for him than us, but I feel bad causing panic moments for people, because I've totally had those moments when I worry I've forgotten some major life change of someone I only see occasionally.