You should have a very good discussion with the guy friend and made it clear to him what you want from him. First, the house if yours and the guy. #1. Told him that he should respect you in your house and not to come and go whenever he likes. #2. You do not want him to com odd time. #3. You are now marriage and you want to remain remarriage with respect as a marriage person. Please stay very firm, because what the guy want is to destroy your marriage.  Encourage him to get marriage if he wants his own wife. You should firm and stop him right away before he move further into your privacy. I would not let something like this to happens to me, because I will raise my voice and  stop him right with first warning and final warning to get his ass together or else I will throw him out of my house. Your husband is a weak and coward person and probably doesn't know what to do and how to stop the guy and try to play along hoping that his friend will know when to stop. They all used to be single and play together as bachelors, but when bachelor became a marriage person. he supposed to set boundary, but he didn't meaning he is immature. Now it is your turn to stand up and stop the stupid guy that doesn't know when to stop. Most bachelor friend will stop on their  own too and respect their friend and his wife.