I definitely think it can look too casual, so you have to dress it up with accessories. Wear a sexy heel and some statement jewelry. And I always find that making your hair look girly helps up the chic factor. Make sure that the chinos and not too slouchy or baggy as it can make you look more sloppy. Also try adding a pretty brooch or bright flower pin to the v-neck!