Red matter was a substance that was created through the use of Decalithium. It was created by the Vulcan Science Academy and took the form of a red liquid substance that collected in the form of a sphere. A single drop of this compound was capable of creating a singularity.
By 2387, the Vulcan government had developed the ability to produce red matter. At the time, the Hobus star had grown increasingly unstable and was going to produce a powerful shockwave that would have destroyed the Romulan homeworld of Romulus. The only way to stop the destruction of the star was by using red matter to collapse it into a singularity. The Romulan Star Empire briefly considered invading the planet Vulcan in order to acquire the technology which the Vulcans refused to share. Ambassador Spock was able to get enough red matter manufactured which was deployed on the Starfleet ship Jellyfish. However, Spock was too late as the supernova from Hobus destroyed Romulus and extinguished the lives of millions of Romulans. Before the nova could progress further, Spock shot a portion of the red matter into the Hobus star collapsing it into a singularity. (TNG comic: "Countdown")
This incident led to the Narada and the Jellyfish falling into the singularity where they went back in time leading to the creation of an alternate reality. During this time, Nero attempted to gain revenge against Spock and sought his emergence from the rift in time where he intercepted the Jellyfish and gained access to red matter. With it, it deployed some of the substance against Vulcan leading to the destruction of the planet and near extinction of the Vulcan species. The crew of the USS Enterprise was later successful in destroying the Narada before it could deploy a red matter weapon against Earth. (TOS movie & novelization: Star Trek)