Burying in a Jewish cemetery
Burying near a Tzaddik, and near a Rasha and near a non-Frum Jew: One may not bury a Rasha near a Tzaddik [and certainly may not bury a Tzaddik near a Rasha]. Furthermore, one may not even bury a small Rasha near a great Rasha. Likewise, one may not bury a Tzaddik and Kosher Jew, and certainly not a Beinoni, near an exceptional Chassid. [The soul of Tzaddik is pained if it is buried near a Rasha. Following this law, one is not to bury an observant Jew near a non-observant Jew, and based on this many cemeteries today contain a Shomer Shabbos section.]
Baal Teshuvah: It is permitted to bury a Baal Teshuvah [who is now a Tzaddik] even near a Tzaddik Gamur. [However a Baal Teshuvah may not be buried near a Chassid.]
Burying a Jew in a gentile cemetery/Burying a gentile in a Jewish cemetery:
Although the Poskim make no specific mention against burying a Jew in a gentile cemetery, or against burying a gentile in a Jewish cemetery, nevertheless, it is included in the above law that one may not bury a Rasha near a Tzaddik. The old age customs dating back to the beginning of our history testify to the establishment of Jewish cemeteries and the particularness of not burying a gentile in a Jewish cemetery, or vice versa. We are thus very particular not to bury a Jew in a gentile cemetery, or vice versa, not to bury a gentile in a Jewish cemetery.
 Michaber 362/5; Mishneh Sanhedrin 46a and Gemara 47a
Is this Biblical, Rabbinical, or a custom? Some Poskim rule that this law is a Biblical obligation, and a tradition of Moshe from Sinai [Chasam Sofer 341; Minchas Elazar 2/41; Igros Moshe 2/152; Teshuvos Vehanhagos as so is learned from the above Mishneh regarding Harugei Beis Din] Others rule that this law is a Rabbinical obligation. [Dvar Mishpat] Others rule that this law is a mere custom. [Maharil Semachos 10]
Other opinions: Some Poskim rule that it is permitted to bury a Tzaddik near a Rasha and the ruling in the above Mishneh is limited to only Harugei Beis Din. [Chasam Sofer ibid in opinion of Rambam who omitted this Halacha]
 Shach 362/5; Taz 362/1
 Story brought in Sefer Chassidim 705 that it is like being buried near a bathroom that has a foul smell; Maharil ibid that the soul of the Tzadik is not revealed the secrets of hevean if he has a Rasha neighbor.
 Igros Moshe Y.D. 2/152; 3/157; Teshuvos Vehanhagos 1/716; 2/598; See Q&A
 Rama ibid
 Shach 362/6 in name of Bach
 See Sefer Umikarev Beyemin 22 for a general overview of this subject; Igros Moshe Y.D. 2/152; 3/157; Milamed Lehoil 2/127; Tuv Taam Vadaas 2/253
 Shulchan Aruch and Nosei Keilim, however see Ritva Gittin 61a that it is forbidden to bury a gentile near a Jew
 Ritva ibid