Djamel Beghal was given a 10-year jail term in 2005 after being sent to France following his arrest in the United Arab Emirates shortly after the September 11, 2001, terror attacks in the US.
He was suspected of leading a network charged by Osama bin Laden to attack American interests in France and is considered by French officials to have been a mentor for several generations of aspiring jihadists.
His activities have also highlighted the struggle by French authorities to prevent Islamic radicalisation in prisons, which have proved fertile recruiting grounds for jihadist fighters.
Beghal, now 52 and stripped of his French nationality, was freed from the Vezin-le-Coquet prison near the western city of Rennes early Monday.
He was brought to Charles de Gaulle airport near Paris for a flight to Algiers, a source close to the case told AFP.
Beghal had been under surveillance for suspected radicalism by French intelligence agents since the mid-1990s, following his arrival in the country from his native Algeria when he was 21 years old.
While serving his first prison sentence Beghal met Cherif Kouachi, one of the brothers who massacred 12 people in an attack on the Charlie Hebdo
magazine in January 2015.
Amedy Coulibaly, the man who killed a policewoman and then four shoppers at a Jewish supermarket just outside Paris that same month, also came under Beghal's influence at the Fleury-Merogis prison south of Paris, where he also met Kouachi.
After their release, both Kouachi and Coulibaly visited Beghal while he was serving out his sentence under house arrest.
Beghal was arrested again in 2010 as part of a plot to free him as well as Smain Ait Ali Belkacem, an Algerian who helped carry out Paris bomb attacks in 1995 which killed eight people.
France has suffered a wave of deadly terror attacks since January 2015 which have claimed nearly 250 lives, prompting the government to make permanent several state of emergency measures as part of a tough new anti-terror law enacted last year.