The report by an independent group of experts said while France had adopted a national action plan to fight racism and anti-Semitism on April 15th, there was lots left to be done.
"The committee is concerned by the recrudescence of hate and xenophobic speech in certain political circles and the media which contribute to the trivialisation of racism and xenophobia," a report said.
It expressed concern over continued racist remarks and commentary on the internet and criticised France for specifically failing to list inciting racial hatred as a crime.
France this year said it was investing €100 million ($110 million) in a major anti-racism and anti-Semitism action plan devised after deadly jihadist attacks in and around Paris in January, which killed 17 people.
The programme, which increases penalties for crimes deemed to be fuelled by racism and anti-Semitism, comes at a time when acts against Muslims and Jews have shot up in France.
The two communities -- the largest in Europe with an estimated four to five million Muslims and around 600,000 Jews -- are not the only targets.
The Roma, a minority group that comes mainly from Romania and Bulgaria, also face considerable discrimination, according to activists.
The experts expressed alarm "at the growing stigmatisation of the Roma" and "the increasing hate speech against them, including by elected political representatives."
In December, a right-leaning French mayor sparked outrage by refusing to allow the burial of a Roma child on the ground that the burial space was reserved for local taxpayers. He later apologised to the family.
The UN panel also denounced "incessant and repeated attacks on their rights to housing" through the forcible clearing of camps without offering them any alternative accommodation.
The evacuations were often accompanied by "excessive police force", there were attacks on the Roma by private individuals, and Roma children were often deprived of even basic education as local authorities often refused to enrol them in schools.
The Roma were also generally unable to access the health care system, they said, calling on France to take measures to protect the community, especially women, from physical attacks.
The report also denounced France's gritty immigrant-filled "ghettos" saying they "can lead to racial segregation."
The experts said they were "especially worried by the fact that these populations continue to encounter discrimination in terms of work, housing, culture, health services and face difficult situations in terms of schooling."
The panel also said the indigenous populations of many French overseas territories -- small islands in the Pacific and Indian oceans -- were discriminated against in terms of social, economic and cultural rights.