Europe is struggling to recover from a devastating economic crisis that has helped fuel anti-immigration and anti-EU sentiment, with far-right political parties gaining traction in several countries, including France.
And with elections to the European Parliament due in May, the migration of workers from poorer to richer member states has become a hot political issue.
"We must have the courage to fight these very negative forces," Barroso told AFP in an interview during a visit to Paris for a major summit focused on security in Africa.
"I am vehemently calling on Europeans to step out of the comfort zone, to quit their silence, to not always let extremes make their move, to have — in France too — the courage to defend Europe.
"I would like it not to be the anti-Europeans who take the initiative by manipulating emotions and anxieties" linked to the crisis and unemployment, Barroso said, less than six months ahead of elections to the European parliament in which extremist parties look well placed.
Last month, Marine Le Pen, head of France's far-right National Front party, and Dutch anti-Islamic leader Geert Wilders announced a "historic" alliance aimed at creating a group at the European parliament after the elections.
This could also include Belgium's Vlaams Belang, Italy's Lega Nord and Austria's Freedom Party. Other countries have strong populist parties, such as Britain's UKIP.
"I would like to see pro-European forces from the centre-left, centre-right and centre have the courage to come and defend Europe, to show that there are things that must be done at a national level too," said Barroso, who in 2014 ends his second five-year mandate at the helm of the Commission.
The former Portuguese prime minister said it was important to "explain that Europe is not the cause of problems but is also part of the solution."
"It isn't Europe that created the excessive deficits and debt of some governments. We're trying to bring a little bit of order… It's Europe that implements regulations and financial supervision that did not exist before at the European level.
"It's not Europe that created confusion in financial markets, it's Europe that is bringing some order," he added, pointing to the record €1.7 billion ($2.3 billion) fine slapped on six big banks this week for rigging key interest rates.
"When we talk about Europe, are we talking about European institutions or decisions that governments are not able to take?" he asked.
As an example, he said the Commission had proposed to tighten a 1996 directive laying out conditions for workers who go to work temporarily in other member states.
"But governments have not yet found an agreement," he said.
The EU directive stipulates that the foreign worker must be given the same working conditions and salary as a national doing the same job, but allows for social security contributions to be made in their country of origin.
Some countries like France warn this has led to "social dumping", with employers taking advantage of the rules to cut costs by avoiding hefty payroll taxes and, in some cases, by paying the migrant workers less than locals.
"It's illegal, it's not Europe that allows that," Barroso said.
The directive is currently being renegotiated and has caused sharp divisions within the 28 member states.
France wants tighter controls, especially on agencies which provide workers from countries like Bulgaria and Romania to northern European countries with higher labour costs.
But that is being opposed by many other countries fear this will restrict the free movement of workers across the European Union — a key principle of the bloc.