The last-64 tie went to an extra 30 minutes when Grenoble came back twice to level at 2-2 before Ghana's Andre Ayew appeared to score the winner with his strike in the first half of extra-time for Marseille.
However a superb header from Grenoble's Selim Bengriba in the dying seconds set up the penalty finale and former Grenoble midfielder Florian Thauvin missed the only kick of the shootout to hand the home side a famous victory.
"We didn't do what we had to do and to lose this match is unacceptable," said Ayew.
"We were leading three times but we now have to concentrate on the second half of the league season which is very important. Grenoble showed a lot of desire and determination but we're very disappointed."
Elsewhere, there were few surprises as seven Ligue 1 sides including Monaco, Lyon and Saint-Etienne all marched into the next round.
Holders Guingamp began their defence in style as they dispatched lower tier outfit Dinan-Lehon 3-2 while Bordeaux edged Toulouse 2-1 in an all top flight affair in the south.
"For our first match after the break, we can be satisfied with our performance," said Bordeaux coach Willy Sagnol.
"We showed something today that was lacking during the last ten days in December and this victory allows us to start the year well and now go to Monaco with some confidence," he added looking ahead to their league match next Sunday at the Louis II stadium.
French champions Paris Saint-Germain take on Montpellier in the only Monday fixture.
Three goals inside the opening 30 minutes was enough for Lyon to hold off Lens 3-2 in Calais while Monaco were too strong for second division Nimes running out 2-0 victors.
"To score three goals away from home in the French Cup is a great performance," said Lyon coach Hubert Fournier.
"We had to be conscious of the mental capacities of Lens who never gave up and it is good preparation for when we face them in the league in the coming weeks."
Franck Tabanou starred for Saint-Etienne when his extra-time volley on 97 minutes was the difference in a 1-0 victory over second division Nancy.
Last year's finalists Rennes, took their place in the next round with a 2-1 win at Dunkirk.
Bastia knocked out Lille on Saturday when Ligue 1 sides Evian, Reims and Nantes all won against sixth division opposition.