A very thoughtful question.
I would say it is OK to speak Swahili/ indigeneous language, and not to speak your language - at least for the given reasons- would constitute an undesirable and wimpish self consciousness, essentially a self esteem question.
The question revolves around two conflicting schools of thoughts, one advocating an individuals right to freedom of expression in their own element/mother tongue together with shunning away cultural imperialism as a negative of globalization, with far reaching iimpacts, as far as the possible extinction of indigenous languages.
On the other hand, polite society has its rules, where one is supposed to be overly concerned about the plight and feelings of others, where perceptions may trump over reality, and frankly, where fear of xenophobia may occupy an even more paramount incentive to speak a global language than politeness.
To me, the tone and loudness of conversation is of more importance than the language.Actually there are some private conversations which I would prefer people to speak in their language if they can.
I am more concerned about the Cantonese cat speaking loud rapid Cantonese because he is speaking loud and rapid, not because he is speaking Cantonese.So if you keep it to yourself, nobody will have a reason to bother you.Who knows, you may even get a Finnish girl interested in Swahili.
On the other hand the fellow insisting speaking English is just another heavy accented African.
I used to have a Scottish eldery co-worker in bongo, every time he would talk to me, in English, I would reply back in English. Then one day he was lamenting his love of languages and how nobody except the cook was speaking to him in Swahili.
Our languages enrich the world we live in, one of my fears (I know it sounds hypo) is that these global languages are going to dominate so much that the indigeneous language will go extinct.This is already happenning in Papua New Guines (albeit Papua New Guinea is a stretch because it is the one country with the most languages per capita in the world, so some of them will die a Darwinian death anyway)
I know the language used above is contrary to the opinions expressed, but the irony is that my love for language fuels my embrace for the not so well known language, English.
On that note I can only be against your friend's position because of the reason he gave for speaking English, and because he was consistently replying in English (Im sure it sounded strange, what was he thinking? By not speaking English the rest of "them" would think you guys were planning to blow shyt up?)