The Olive branch was a symbol of peace or victory. It was used mostly by the Federation when they sent envoys on missions of peace to various planets and governments.
In the 2261 of the Alternate reality, Capt. James T. Kirk decided to extend a olive branch towards the klingons when sent a distress call from Khitomer when the Romulans attacked. (TOS - The Khitomer Conflict comic: "Part 1")
In 2293 of the Prime reality, Capt. Kirk and his crew of the USS Enterprise-A were then chosen by the C-in-C to act as the Olive Branch to support Klingon Chancellor Gorkon's peace initiative.(TOS movie, novelization & comic adaptation: Star Trek VI: The Undiscovered Country)