I just had to use extra virgin olive oil in my banana bread yesterday (was out of vegetable oil and didnt realize). Honestly it turned out just as delicious. I did add about a teaspoon of vanilla (which the recipe didnt call for). But very tasty and will definitely use again!
Yes you can use olive oil for banana bread instead of vegetable oil but the taste will be different from both so try out both for sample so that you can check which one is better for you.
I believe that olive oil is more for cooking and vegetable oil for baking. If you want something healthier than vegetable oil then you may want to give canola oil a try when you're baking.