To put it in very simple terms, the flower is the parent of the fruit. You don't get fruit without flower. If you look very carefully at the stem of the fruit, where it connects to the fruit itself, rather than to the branch, you will see what remains of the flower. It's often dried and looking pretty used up, but usually recognizable if you already know what it is.