"Two victims have been stabbed to death," regional police chief Olivier de Mazieres told AFP, referring to the attack which occurred at 1:45pm (1145 GMT).
Local prosecutor Xavier Tarabeux said the knifeman had been killed by soldiers, while the Marseille police urged people in the city to avoid the area around Saint-Charles station in the bustling centre of the city.
The knifeman is believed to have shouted "Allahu Akbar" (God is Greatest) before assaulting passers-by, a source close to the investigation told AFP on condition of anonymity.
The deaths came with France still on high alert following a string of terror attacks since January 2015, when jihadist gunmen stormed the offices of the satirical weekly Charlie Hebdo, killing 12.
The government has since launched Operation Sentinelle, deploying about 7,000 troops across the country to guard high-risk areas such as transport hubs, tourist sites and religious buildings.
After the stabbings in Marseille, anti-terror prosecutors said they had opened an investigation into "killings linked to a terrorist organisation" and the "attempted killing of a public official".
Attacks by Islamist extremists since 2015 have left 239 people dead in France, according to an AFP count before Sunday's incident.
New anti-terror law
After a rampage by Islamic State gunmen through Paris in November 2015, Francois Hollande, president at the time, declared a state of emergency which remains in place, giving security forces greater powers to use force and launch anti-terror raids.
Hollande's successor Emmanuel Macron has vowed to end the state of emergency with a new and controversial security law that will make many of the provisions of the state of emergency permanent.
Despite criticism from rights groups, the lower house of parliament is set to vote on a first draft of the law on Tuesday.
French Interior Minister Gerard Collomb posted on Twitter that he would travel to Marseille immediately.
As well as the major terror atrocities in France in recent years - Charlie Hebdo, the November 2015 Paris assault and a killing spree in Nice in July 2016 - there have been a series of smaller incidents.
Attacks on police officers, soldiers or members of the public with knives, firearms or vehicles have sometimes been carried out by people with severe psychological problems.
The incident in Marseille came only days after the Islamic State group released a recording of what it said was its leader Abu Bakr al-Baghdadi urging his followers to strike their enemies in the West.
France has deployed troops and its air force to the Middle East and is a leading partner in the US-led international coalition fighting the Islamic State group in Iraq and Syria, where the jihadists are being driven back.
In August, a man driving a van killed one person and seriously injured another after ploughing into a bus stop in Marseille, raising fears of another terror incident.
But doctors said later that the man had severe mental problems and discounted any terror link.
By Francois Becker with Adam Plowright in Paris