It's not in the "plant-based diet" if it contains animal fats or animal protein, or even plant based saturated fats. Highly processed foods (such as anything containing corn syrup and refined sugars) also to be avoided. I violate the rules by still having milk in my coffee (not every day anymore) and a small portion of sweets now and then. Giving up meat was easy, but giving up cheese was more difficult (I love Mexican and Italian foods, and there isn't much on those menus without meat or cheese). It's caused a shift toward Indian and Asian foods and sauces/spices for my wife and I, and we're exploring recipes and cooking different foods we never knew about before. Interesting and many are spicy -- no cardboard food for us. Been doing this since November and plan to continue. If we ever start to eat meat again it will be in much smaller amounts than in our past.
Gotta say I do feel different (better). Animal fats (and other saturated fats including non-animal fats such as coconut oil) have long been linked to cardio pulmonary disease and stroke. I think getting rid of saturated fats really makes a big difference in how I feel, although dairy is also a big issue and most people don't know it because we've all grown up thinking it's a "given" that milk is good for us.
Cow’s milk is only good for you if you’re a calf. Aside from lactose which causes at least mild gastric reaction in many people, the milk protein (casein) can cause many issues. It has been linked to food allergies and auto-immune diseases including Type 1 diabetes ( http://www.greenmedinfo.com/blog/confirmed-anew-cows-milk-may-trigger-type-1-diabetes ) and MS ( http://mssociety.ca/en/research/PC010321.htm ) but not limited to that.
I've learned you don’t need milk for calcium, and don’t need meat for protein. Your body uses amino acids to make protein and you get plenty of them and calcium, vitamins and minerals from plant sources, along with anti-oxidants to fight cancer (you don’t get those from animal sources).(nl)