Minhagim

So, my wife and I have been having this debate (even though we're both pretty much on the same side of the debate) about minhagim and the status of many minhagim as actual halacha. It seems some minhagim can just be chucked, while others we're stuck with forever.

For example: I grew Lubavitch. Around the time Artscroll published its first all Hebrew siddur, I decided enough was enough and chose to no longer be a Lubavitcher (a decision I have NEVER regretted in any way). So, I stopped keeping Chalav Yisrael (a concept at which the Yeshivish community then scoffed and now holds to be unbreakable halach to KEEP this chumra in the US). I started davening Nussach Ashkenaz. I stopped putting on Rabbeinu Tam teffilin (do you ANY idea what a pain in the rear THAT was?). My rabbanim then and since then have never had a problem with this. In fact, they felt I had finally gotten on the correct path and were more than happy to encourage me in this direction.

So, how are those minhagim different from others, like waiting six hours between eating meat/poultry and dairy? Or not eating Kitniyos on Pesach? Or, to suggest an even greater extreme, keeping two days of Yom Tov? No one seems able to give me a straight answer. It seems that, because my dad, when he became a Ba'al Teshuva, took on the chumrah of waiting six hours, that I'm now stuck with this chumrah/minhag as well and have no choice but to continue this idiotic "mesorah" of waiting six hours between meat and dairy. How is this any different than the other I mentioned above? I can never seem to get a straight answer, or more of one than, "well, that's what your family does." When my wife became a Ba'alas Teshuva, she kept six hours, not realizing she had a choice. Now, she's stuck too.

So, why can't I chuck minhagim with which I'm not comfortable? It's not halacha. In fact, the Gemara mentions several instances of how long certain rabbanim waited between meals, and it had to do with how often they ate, not how long they felt they should wait.

I just don't get it...