The uproar started when the Icéo pool in the northern French town introduced a measure which effectively prevented refugees from entering the pool.
Anyone who wants to visit the pool, located around 20 minutes walk from the squalid Jungle camp, must now show an identity card and proof of address, which the vast majority of refugees do not have.
Rights groups have reacted angrily to the measure that was imposed in the wake of the Paris terror attacks.
Changing the conditions of access is “a discriminatory offence that only targets a population already highly stigmatized,” said Claire Rodier from immigrant support group Gisti.
Jean-Pierre Alaux, from Gisti told The Local that the swimming pool is not the only place in Calais where migrants and refugees have been banned.
"Over the last 15 years we have seen bars and shops block foreigners from entering and library did the same," he said.
"There have been many scandalous incidents here and we had to try to do something," he said.
In a letter sent to the French prosecutor and to France's top independent rights body the Defenseur des droits 19 rights groups and aid associations, including Amnesty International, denounced “the racial discrimination” behind the move.
They demanded authorities investigate the legality of the measure that is "manifestly discriminatory and clearly contrary to law”.
The outspoken mayor of Calais Natacha Bouchart has defended the move on the grounds of the need to guarantee security.
She says the measure was taken under the ongoing state of emergency in France to target “people whose identity we don't know”.
Security has been increased at public buildings all around France in the aftermath of the terror attacks. Security guards have been brought in to carry out bag searches and scans of members of the public.
While demanding to see and ID card is common in France, where the public are obliged to carry one at all times, asking for proof of address is extremely rare.
Bosses of the municipal pool were refusing to comment on Thursday.