The occupation started before dawn when demonstrators entered the site on the outskirts of the city where a large grey building with a minaret is being built.
Climbing on to the roof they unfurled two banners, one identifying their organization, the other proclaiming "Charles Martel beat the Arabs at Poitiers in 732."
That battle is credited with having halted the advance of Islam into western Europe.
Towards noon the activists agreed to leave and the evacuation was completed by 1pm.
"We were planning to stay longer but as we had no intention whatsoever of a physical confrontation we are leaving with the police in a good mood without an unhappy ending," a spokesman for the demonstrators, Damien Rieu, said as the occupation ended.
Four of the occupiers who said they were the organizers were held for questioning, the prosecutor's office said. The rest underwent identity checks.
French Prime Minister Jean-Marc Ayrault, who is in the Philippines, "firmly" condemned a "provocation that reveals an unacceptable religious hatred."
Interior Minister Manuel Valls also condemned "hateful and inadmissible provocation" and the "questionable confusion" of the group.
High profile anti-racism groups Movement against Racism and for Friendship between People (MRAP), SOS Racism and the Union of Muslims in France (RMF), also called for extremist groups to be banned and their websites to be shut down.
Far-right National Front leader Marine Le Pen expressed shock on Monday at the government’s “disproportionate” reaction, and the call to dissolve the extreme-right youth group behind the attacks, Génération Identitaire.
“I was taken aback by the hysterical reactions of the political class,” she said on radio station RTL.
“I would like to have seen the same reactions of hysteria for the multiple church occupations.
“I understand the fears and the worries over the construction of these giant mosques, these cathedral mosques with 22-metre-high minarets, being put up often without asking the opinion of the locals.”
The police have opened an inquiry into "an unauthorized demonstration, incitement to racial hatred, participation in a gathering with a view to carrying out joint damage."
There could be charges of theft as about 10 carpets were removed from the mosque and taken to the roof where they were badly damaged by rain.
"Poitiers is in shock," Alain Claeys, the mayor of the town of 90,000 inhabitants said.
Extreme right-wing activists had come from all over France and seemed well organized with banners and computers, he said.
Yves Dassonville, the prefect of the Vienne region in which Poitiers is situated, told reporters that the 73 demonstrators were "seemingly people of good family who came from all over France."
The Muslim national umbrella organization, the French Muslim Council (CFCM), spoke of its "strong indignation" and "its deep concern in the face of this new form of anti-Muslim violence."
The occupation, it said, had been "serious, savage and illegal, accompanied by slogans hostile to Islam and without precedent in the history" of France, which has the largest Muslim population in Europe, of between four and six million.
Local Muslim leader and imam Boubaker El Hadj Amor said his local community of between 7,000 and 8,000 people was deeply hurt.
He praised its members for staying calm and refusing to rise to the provocation.
Left-wing groups called for the group responsible for the occupation to be dissolved.
The International League against Racism and Anti-Semitism (Licra) called the occupation a "demonstration of hate."