The answer you're looking for is fruit, and more specifically a 'stone fruit' - and thus in the same group as cherries, apricots, plums etc...
To fruit or not to fruit?
The key to telling whether something is a fruit or not is that fruits are specifically the part of a plant that 'disseminates', or carries and disperses, the seeds.
So in the case of the humble olive, the pit or seed is surrounded by a 'flesh', which acts as the vessel.
In botanical terms, the definition to go by is that fruits are "a part that developed from the ovary of a flowering plant".
Vegetables, on the other hand, derive from parts of the plant that aren't directly related to carrying and disseminating seeds.
Confused between fruit and vegetable?
Whilst distinguishing between fruit and vegetable might seem fairly straight forward, it's actually really not!
Botany, culinary tradition, cultural tradition and taxation laws all have different definitions on what constitutes a fruit.
For example, bell peppers and egg plant (aubergine) is technically a carrier of seeds, and thus a fruit - but you'd be hard pressed to find a supermarket that would place them in the same aisle as apples or bananas!