SOS Racisme denounced as "the most anti-Semitic filth" a pledge by the founder of the National Front (FN) party to put his critics in their place using a pun suggesting Nazi gas chambers.
The remark was contained in a video clip posted on the FN website and since removed.
In the video, Le Pen railed against a number of critics including pop star Madonna and Yannick Noah, the French singer and former tennis champion.
When asked about another one — French singer Patrick Bruel, who is Jewish — Le Pen said he would be part of "a batch we will get next time," using the word "fournée" for "batch", evoking the word "four", which means "oven".
SOS Racisme said it would file a complaint "in the coming days" against Le Pen, whose daughter Marine took over as FN leader in 2011.
She has sought to rid the party of its reputation for racism and anti-Semitism in recent years, and members were quick to distance themselves from the founder's comment.
FN vice president Louis Aliot, who is also Marine's boyfriend, criticized it as "a bad phrase. It is politically stupid and dismaying."
Another FN lawmaker, Gilbert Collard, said it was time for the party founder to "take his retirement".
But the 85-year-old Le Pen reacted by saying: "If there are people in my camp that have interpreted it in this way, they are nothing but imbeciles."
He denied any anti-Semitic overtones in the remark.
"The word 'fournee' that I used obviously has no anti-Semitic connotation, except for political enemies or idiots," he said.
His daughter gave a guarded response saying that her father had made a "political error" but that the "meaning given to his comments is a malicious interpretation."
Another group, the Movement Against Racism and for Friendship Between Peoples, also said it would lodge a complaint, calling Jean-Marie Le Pen "an authentic anti-Semite".
Le Pen, who has had multiple convictions for inciting racial hatred and denying crimes against humanity, once described Nazi gas chambers as a "detail" of history.
Le Pen has used similar wordplay in the past to make light of gas chambers used by the Nazis in World War II. In 1988, he referred to a Jewish minister named Michel Durafour as "Durafour-crematoire". In French, the word "four-crematoire" means crematorium.
The FN party triumphed in European Parliament elections last month, winning 25 percent of the vote.