Interior Minister Manuel Valls and Najat Vallaud-Belkacem, Minister of Women's Rights, were due to go to Thursday's meeting in Florence that will also be attended by European Commission head Jose Manuel Barroso.
They "decided not to attend this conference... at the same time as Tariq Ramadan," an aide of Vallaud-Belkacem, who is also spokeswoman for the Socialist government, told AFP.
The aide said the two ministers had not wanted their presence to be interpreted as France endorsing Ramadan, "who supports obscurantist positions contrary to the values of the Republic."
Ramadan is a Swiss academic who is professor of contemporary Islamic studies at Oxford University.
He is the grandson of Hassan al Banna, an imam who in 1928 founded the Muslim Brotherhood in Egypt, which carried out bloody attacks in the 1940s.
His views on Islam have attracted controversy, and he has in particular been accused of defending the stoning of adulterers, a death sentence still meted out in some countries including Saudi Arabia.
Ramadan however has said he does not condone stoning and instead calls for a moratorium on the practice, to enable countries involved to have a debate on the issue. This too has been criticized for not going far enough.
The conference, dubbed "State of the Union", is organized by the Florence-based European University Institute, a teaching and research organization focused on studying Europe.