Kovac, a former Bayern midfielder, led the side to the league and cup double in May, becoming the first man to achieve the feat as a player and manager, but things have turned sour this season and a decision has now been made to part ways.
A statement on the club website said the decision was by mutual agreement, with the Bundesliga champions down in fourth place having lost 5-1 to Eintracht Frankfurt on Saturday.
Kovac said on fcbayern.com: "I think this is the right decision for the club at the moment. The results, and also the way we last played, made me come to that decision.
"My brother (and assistant) Robert and I thank Bayern for the last one and a half years. During this time, our team has won the championship, the DFB Cup and the Supercup.
"It was a good time. I wish the club and the team all the best."
Assistant coach Hans Flick has been placed in temporary charge ahead of Wednesday's Champions League clash against Olympiakos and Saturday's Bundesliga meeting with Borussia Dortmund.
Chief executive Karl-Heinz Rummenigge said: "The performance of our team in recent weeks and the results have shown us that there was need for action.
"(President) Uli Hoeness, (sporting director) Hasan Salihamidzic and I had an open and serious conversation with Niko on this basis on Sunday, with the consensual result that Niko is no longer coach of Bayern.
"We all regret this development. I would like to thank Niko Kovac on behalf of FC Bayern for his work, especially for winning the double this past season."
Bayern sit four points behind early league leaders Borussia Monchengladbach with five wins and 18 points from 10 matches.
Former Juventus coach Massimiliano Allegri and Jose Mourinho are among the early names in the frame to replace him.