Cazeneuve told the state-owned radio France Info: "It is not normal that countries like Austria for example do not communicate information they have regarding European Union citizens who have accounts there.
"If these countries do not cooperate, if there is not a convention covering an information exchange that allows for total transparency within the European Union, these countries run the risk of being on the list of non-cooperative states and territories," Cazeneuve said.
Austria is the last EU member to resist lifting bank secrecy policies, after Luxembourg said Wednesday that it would implement rules allowing for the exchange of account information starting in 2015.
Cazeneuve was named French budget minister on March 19 to replace Jerome Cahuzac, who resigned as it emerged that he had maintained a non-declared bank
account in Switzerland.
On Wednesday, President Francois Hollande unveiled a series of proposals aimed at reinforcing transparency in the banking sector and said that France would establish a list each year of tax havens.
Austrian Chancellor Werner Faymann huffed that his country has no need for "unsolicited advice from the outside" in comments to appear in Friday's edition of the daily Kurier.
"We will find a solution concerning the exchange of data," he reassured the daily. "We must put fighting tax evasion at the top" of our priorities.
A French list of non-cooperative countries and territories (NCST) has in fact existed since 2010, and its latest update in April 2012 contained eight names, down from 18 in the original version.
At present, Botswana, Brunei, Guatemala, the Marshall Islands, Montserrat, Nauru, Niue and the Philippines are considered by Paris to be uncooperative in the fight against tax fraud.
Leading European countries have called meanwhile for a continental version of the US Foreign Account Tax Compliance Act (FATCA), which obliges foreign countries to reveal information on the investments and revenues of US citizens abroad.