An agreement "will not be accepted by France if it is not clear that verifications can be made at all Iranian facilities, including military sites," Fabius told parliament.
Iran's supreme leader Ayatollah Ali Khamenei last week ruled out inspections at military sites.
But Yukiya Amano, the head of the UN's atomic watchdog, told AFP on Tuesday that Iran has agreed to implementing the Additional Protocol of the Nuclear Non-Proliferation Treaty (NPT) that allows for snap inspections of its nuclear facilities, and if required, military sites.
"When we find inconsistency or when we have doubts, we can request access to the undeclared location for example, and this could include military sites," said the Japanese diplomat.
"Some consideration is needed because of the sensitiveness of the site, but the IAEA has the right to request access at all locations, including military ones."
But Iran appears to be interpreting the protocol differently. As well as Khamenei's comments, Foreign Minister Mohammad Javad Zarif has said the protocol allows "some access" but not inspections of military sites, in order to protect national "military or economic secrets".
Iran and the so-called P5+1 group -- Britain, China, France, Russia, the United States plus Germany -- have been engaged for nearly two years in negotiations on Tehran's nuclear programme.
The deal is aimed at preventing Iran from developing the atomic bomb in exchange for an easing of crippling economic sanctions.
The two sides signed a framework agreement on April 2 and began meeting in Vienna on Wednesday to start finalising a deal which is due by June 30.