Ayrault's comments came after the country's leading actor, Gérard Depardieu, took up residence in a tiny village just over the border in Belgium which is a favoured spot for wealthy French nationals avoiding tax.
"Those who are seeking exile abroad are not those who are scared of becoming poor," he told reporters after announcing sweeping anti-poverty measures to help those hit by the economic crisis.
He said they were doing it "because they want to get even richer," adding: "We cannot fight poverty if those with the most, and sometimes with a lot, do not show solidarity and a bit of generosity."
"Thankfully, few are seeking exile to exempt themselves from being in solidarity with fellow Frenchmen," he added.
A lawmaker with the ruling Socialist Party, Yann Galut, suggested parliament should adopt a law depriving exiles of French citizenship if they pay no taxes in France.
"Failing a settlement of the difference between the taxes paid in the country that hosts them and the taxes they would have paid in France, these tax exiles would be stripped of their French citizenship," Galut said.
France's Socialist President François Hollande, who famously declared "I don't like the rich" during his election campaign, has pledged to tax annual income of more than one million euros per year at 75 percent.
British Prime Minister David Cameron ruffled feathers across the Channel earlier this year when he said he would "roll out the red carpet" for French residents trying to avoid the top tax rate.