The Conseil d'Etat (State Council) says that the presence of the cross goes against the 1905 French law which officially separates church and state.
The town of Ploërmel in the northwestern region of Brittany now has six months to remove the cross.
However the statue of the former pope can remain because, according to the council, the effigy of the pontiff "cannot, in itself, be regarded as a religious sign or emblem".
"As the cross constitutes a religious sign or emblem as defined by Article 28 of the law of December 9th 1905 and its installation by the local authorities does not fall into any of the exceptions provided by this article, its presence in a public location is against this law," said the Conseil d'Etat in a statement.
Politicians on the right and extreme right were quick to denounce the decision as an attack on France's cultural heritage.
MP Valérie Boyer for right-wing Les Republicains party said: "Where will we stop this madness of wanting to erase our roots?"
"Remove religious symbols from the public space, our streets, our cemeteries? A society without soul and without history? " said Roger Karoutchi who represents Les Republicains in the French senate.
As for Louis Aliot, vice-president of the far right National Front party, he called the move "an unfair decision" which "participates in the work of destruction of our Judeo-Christian civilization".
And the row has even reached the ears of the Prime Minister of Poland, where John Paul II was born, who has offered to give the work a home to save it "from censorship".
The argument also played out on Twitter with people using the hashtag #MontreTaCroix (Show your Cross) to post photos of religious crosses in defense of Ploërmel, whose mayor has fought long and hard to preserve the cross.
Some said they feared that this would mean that crucifixes would start disappearing from churches and cemeteries.
However Jean-Louis Bianco, an expert on secularism, said this fear is "unfounded", with Article 28 of the 1905 law providing for several exceptions to the presence of religious symbols in the public space.
They are permitted on "buildings of worship, burial grounds in cemeteries, funerary monuments, and museums or exhibitions," he said.