Project Genesis (also called the Genesis Project) was a scientific research project focused on creating inhabitable planets out of uninhabitable ones. Project Genesis was named after Genesis, the first book of the human bible, in which God created Earth and the human race. The project was led by Dr. Carol Marcus and her son, Dr. David Marcus.
In the 2280s the team created a Genesis matrix in a cavern on Regula, and designed a prototype Genesis torpedo. While the matrix placed in the torpedo did have the information programmed in to it to create an entire star system, it was intended that the device would be used on a lifeless planetary body. (TOS movie, novelization & comic adaptation: Star Trek II: The Wrath of Khan)
This torpedo was stolen and later detonated by Khan Noonien Singh in the Mutara Nebula. This detonation created an entire star system, complete with a class M planet. However, protomatter was used, and the resulting planet was not stable, and exploded only a few weeks after its creation. At the time protomatter was blamed for the failure of the Genesis project. (TOS movie, novelization & comic adaptations: Star Trek II: The Wrath of Khan, Star Trek III: The Search for Spock)
Some time after the destruction of the Genesis Planet, Starfleet conducted an analysis in which they concluded that the planet was unstable and exploded not because of the protomatter used, but instead because of the nature in which the device was used. The device was meant to explode on the surface of a dead planet - and the resulting Genesis Wave would be able to concentrate on rebuilding the planet. If it had been used in that manner, even with protomatter being used, the planet would have been quite stable.
However, the prototype was detonated on board a starship, which was in a nebula of diffuse and highly charged gaseous particles. As a result, the Genesis Device first had to build an entire star system - complete with a G-type star - before it could create a class M planet. After building a star system, the Genesis Wave did not have sufficient energy to complete the building of the Genesis Planet. Also, the device created two nearby gas giants, which created additional gravitational stresses on the planet. (TNG novel The Genesis Wave, Book 1).
In the Mirror Universe, Project Genesis was canceled by Emperor Spock when it reached stage two. However, stage two became the cornerstone for Memory Omega, a project the Emperor had created to save the collective knowledge of the Terran Empire. (Star Trek: Mirror Universe novel: The Sorrows of Empire)