Scolari is back in west London for Brazil's friendly against Fabio Capello's Russia on Monday and he took the opportunity to set the record straight about his seven-month reign as Blues boss.
At the time of Scolari's dismissal in February 2009, Chelsea were fourth in the Premier League and still in both the FA Cup and Champions League.
That wasn't deemed good enough by Chelsea owner Abramovich, who reacted to complaints from players about what they perceived as Scolari's overly-relaxed management style by axing the coach.
Scolari feels his Chelsea reign has since been wrongly written off as a failure and he tried to put a more positive spin on it at the pre-match press conference before Monday's friendly.
"I believe my time at Chelsea was not as bad as people have been talking about," Scolari said.
"The numbers show that, the games we won. I don't think it was that bad."
Asked whether he was given enough time, Scolari added: "I don't think so but it was a decision from the management.
"Everywhere I work I am just a member of staff and if somebody thinks I am not doing a good job, they can sack me.
"That happens everywhere. I accept that. I did enjoy my time here at Chelsea and I was happy while I was here. It was one of the great experiences I have had in my life."
Scolari was replaced by Guus Hiddink, who won the FA Cup, finished third in the Premier League and reached the Champions League semi-finals before a controversial defeat against Barcelona.
Chelsea's Brazil defender David Luiz claims Scolari is still held in high regard at the club, but admits he could have a point to prove to Abramovich against Russia.
"Mr Scolari is an excellent professional. He has been a coach in many clubs all over the world and also national teams," Luiz said.
"I'm sure he is happy to come back here because regardless of what happened he has very good memories.
"He has got history here as well so I believe what he wants is to put the cherry on the top of the cake by winning the game at the stadium where he was manager.
"When somebody doesn't do well everybody outside always talks about the negatives not positives.
"What I have heard, having asked around, is Scolari had a great time, regardless of what happened."
Chelsea are arguably in a worse position now than when Scolari left, having become the first defending champions to be eliminated in the group stages of the Champions League.
They are also 19 points behind Manchester United in the Premier League, while interim manager Rafael Benitez is widely despised by Chelsea fans.
Luiz recognises it has been a winter of discontent but he is convinced Chelsea can still rescue their season.
"I think we are not playing at the level expected from Chelsea and not doing so well this season but in any case I believe we are because we are still competing for three titles," Luiz said.
"We can still win the Europa League, are still in the FA Cup and fighting for a better position in the table so I think we can still get there and this season can still be very good."
What the stars mean:
★ Poor ★ ★ Promising ★★★ Good ★★★★ Very good ★★★★★ Exceptional