Prime Minister David Cameron's office on Monday warned that Britain leaving the European Union could mean migrant camps being set up on British shores if UK border checks are removed from Calais in France.
"Should Britain leave the EU there's no guarantee those controls would remain in place," Cameron's spokesman said at a daily briefing.
"If those controls weren't in place there would be nothing to stop thousands of people crossing the Channel overnight and arriving in Kent (southeast England) and claiming asylum," he said.
Cameron will use the argument in the weeks after he calls a referendum date, according to reports.
Cameron's spokesman said Britain and France had "positive working relations" on border issues.
"The point being made here is that a potential departure from the EU throws that whole relationship into question," he said.
Campaigners for Britain to leave the 28-country bloc accused the government of "scaremongering."
"The agreement we have to process migrants in Calais is with France, not the EU. There is no reason for this to change on leaving the EU," said Arron Banks, co-founder of the Leave EU campaign.
"Having failed to deliver the fundamental reform of the European Union that he promised... the prime minister is now resorting to scaremongering," he said.
But the pro-EU Britain Stronger in Europe campaign said leaving would impact the border.
"Britain's membership of the EU means we have the best of both worlds," said Will Straw, executive director of the group.
"Thanks to an agreement with the French, our border controls are located in Calais rather than Dover.
"If we left the EU, there is a very real possibility that arrangement could come to an end, making Britain's borders less secure," he said.
The migrant crisis in Calais has meant there have already been repeated calls in France to move the border the UK border back, whether Britain leaves the EU or not.
Many in France are now looking back at the 2003 treaty with incredulity that former Interior Minister Nicolas Sarkozy ever agreed to sign it.
Although the agreement was reciprocal, there are not thousands of migrants trying to get from the UK to France, hence the feeling that France was hard done by.
Calls for France to scrap the treaty have previously been made by the outspoken mayor of besieged Calais, former ministers as well as by France’s national Commission for Human Rights.
“This tragic situation is largely the consequence of the signing of several bilateral treaties and arrangements between France and the UK,” said the commission in a statement last year.
The commission added that “tangle of treaties”, which are largely in contradiction of EU law “leads to France becoming the police arm of British migration policy.”
According to professor Pierre-Yves Monjal, a public law specialist from the University of Tours, France does have the legal power to say stop, even if Britain didn't quit the EU.
“France could use article 56 of the Vienna Convention by applying the general EU rules (on terminating a treaty)," he said.
The last article of the treaty of Touquet grants the two signatory powers the option of ending the treaty “at any time”, simply by informing the other party in writing, via diplomatic channels. There would however be a two year delay before the change came into effect.
France also has powers to make changes to the treaty by a “simple exchange of notes” as well as taking the more drastic measure of employing its right to “take all necessary measures to safeguard its sovereignty and security”.
But despite Monday's statement from Cameron's office, many don't believe France would seek a border shift, as it probably wouldn't alleviate the crisis that has seen thousands of migrants amassed along the northern coast.
If the French ever told the UK to move back its borders, there would still be an urgent security crisis in Calais, except the British would likely withdraw their financial help that amounts to millions of euros.
In September last year London agreed to hand over an extra €15 million over three years to help deal with the crisis in Calais.
And last month Home Secretary Theresa May said another €7 million would be made available to increase security at the port and Eurostar terminal in Calais and only this week the UK said it was sending 100 extra border guards across the Channel.
And with the extra border checks and UK investment in security ever goes, the fear in France is that even more migrants would head to Calais if they thought it would be easier to get to Britain.
Last October France's Interior Minister Bernard Cazeneuve said: "Calling for the border with the English to be opened is not a responsible solution.
"It would a send a signal to people smugglers and would lead migrants to flow to Calais in far greater numbers. A humanitarian disaster would ensue. It's a foolhardy path and one the government will not pursue."
Despite the growing opposition and anger in France and the fears in Britain of Paris taking advantage of any Brexit it seems the British border police are in France for the foreseeable future.