The CFCM "is concerned over the direction the public debate is taking," the president of the body, Anouar Kbibech, said in a statement calling for an urgent meeting with Cazeneuve, citing the "growing fear of stigmatization of Muslims in France".
The interior ministry later announced the meeting would take place on Wednesday afternoon.
Kbibech noted that a few days ago a woman was fined on a beach in Cannes while wearing a simple headscarf - which is not included in the ban - and accompanied by her children.
The 34-year-old mother, who gave her name only as Siam, said she had been sitting on the beach in leggings, a tunic and a headscarf, when she was
fined. She said she felt she had been targeted "purely for being Muslim" and was also racially abused by fellow beachgoers.
The CFCM said it was also worried about the photos that have emerged of a Muslim woman removing her tunic on a beach in Nice while surrounded by four armed police officers.
The images, which have been shared around the world on social media, have widely been interpreted as the woman being ordered to remove her tunic by police, although the circumstances remain unclear. She was wearing a sleeveless shirt underneath.
"We have seen images of police officers forcing a woman on a Nice beach to remove her tunic when she wasn't even wearing a burqini," said the statement from the CFCM. "With the difficult and critical situation France is facing after the tragic attacks which deeply affected the country, the CFCM calls for wisdom and responsibility from everyone."
"Today, we need more acts of peace and tolerance," said Kbibech. France's highest administrative court, the State Council, will on Thursday examine a request by the Human Rights League (LDH) to scrap the bans adopted by some 15 towns and cities across the country.
Lower courts have supported the decision by French mayors, with a tribunal in the Riviera city of Nice - where a crowd was mowed down in July in a grisly truck attack - said the burqini could "be felt as a defiance or a provocation exacerbating tensions felt by" the community.