Please never argue with him or never give him any reason to feel he that you ere jealousy, I am married and I know the value of beautiful decent women n a mother, they are hard to come.
Your value and image at your current position is very high, usijiangushe hata kidini.
Also call your collegue in the office face him with courage, tell him you respect him for admiring you, but he should be decent to respect other pls wife. Kama unaweza muulize JEE angefurahiya kumuona mama yake akimdhulumu babayake, atashtuka ama kukasirika, than tell him that's what I don't want for my children.