What is one to do if he switched Talleisim with another person?
The general law by a switched item: One who [accidently] took the wrong item may not use the item [that he accidently took] and doing so is considered stealing. The same applies if one’s item was accidently taken and he then desires to take the item of the person who took his item, he may not use the item and doing so is considered stealing. This prohibition applies even if one knows that the other person is [stealing and] using his item [without his permission], he may nevertheless not make use of his friends item [unless he receives explicit permission from him]. Upon the rightful owner of the item being found the person must return him the item even if his own item became lost [not due to the fault of the rightful owner].
B. The law:
If one switched his Tallis with another person, he may use the Tallis as if it were his until he is able to switch it back with the person who took his Tallis. Upon being asked to switch back, one must do so.
 Admur Gzeila Ugineiva 30; Michaber C.M. 136:2; Bava Basra 46a
 The reason: As even if in truth his friend is using his item, if he were to use his friends item in exchange [without receiving his explicit permission] it is considered as if he is stealing from a stealer, and it is forbidden for one to place judgment alone without a Beis Din, as was explained above in Halacha 27. [Admur ibid; See however Aruch Hashulchan 136/2 who seems to learn that if one knows that the owner is using his item then he may use the owners item even without his consent. Admur ibid clearly negates this approach.]
 Admur ibid; Rama ibid; Terumos Hadeshen 319
 14:9-11; Michaber 14:4; Ketzos Hashulchan 7:8
Admur 14:9; Michaber 14:4
The reason: This is not considered as if one is borrowing without permission, which renders one a thief [Gazlan], being that there is no worry that the borrowed item will get worn out [Kiluiy Karna], as one is only wearing it for a short amount of time and then returning it to him afterwards. Now, being this is the case, we apply the rule that a person is pleased that a Mitzvah be performed with his object when it does not incur him any loss. [Admur ibid; See also 437:7 regarding Bedikas Chametz; 586:5 regarding Shofar; 637:9 regarding Sukkah; 649:3 regarding the Daled Minim; Admur Choshen Mishpat Shiela Usichirus 5 and Metzia Upikadon 27; Kol Yaakov 691:36 regarding Megillah]
Other Poskim: Some Poskim rule one may not borrow a Tallis without permission being it is not an obligation to wear a Tallis, and this allowance only applies to Tefillin and a Lulav. [Shulchan Hatahor 14:3] The Poskim negate his opinion. [Maharsham 1:29; Chazon Nachum 2:1; Hisorerus Teshuvah 1:6] See next footnotes for opinions that hold that today the above allowance no longer applies being that people are particular against others using their Tallis. Some Poskim rule the above allowance only applies if one found the Tallis in the shul or Beis Midrash, and not in a private area. [Aruch Hashulchan 14:12; Igros Moshe 5:20] However from Admur below “One may not take it from that area to a different area, such as to take the Tallis from the Shul to his home or from the owner’s home to Shul.” It is evident that even in a private area it may be used.
The law if one is already wearing a Tallis Katan: Some Poskim rule that if one is already wearing a Tallis Katan, he may not borrow the Tallis Gadol without permission. [Makor Chaim of Chavos Yair 14]
 Birchas Habayis 37:10; See Kesef Kedoshim 126
 The reason: Although we rule that when one switched an article with another it is forbidden to use it due to stealing [Admur Gzeila Ugineiva 30; Michaber C.M.136:2] nevertheless by a Tallis since we assume each person does not mind if the other uses his Tallis until they switch it back, as even in a case that the Talleisim were not switched one may use his friends Tallis without permission under this basis, and thus certainly in this case it is permitted. [ibid]