Remember a goat is a livestock animal and just like a horse it is in need of a companion. A single goat will cry and call a lot and quite possibly all day long, may attempt and even escape from its enclosure. Death to the animal may result from eating something it shouldnt (read more on poisonous plants), getting hit by a car, being injured or killed by a neighbors dog etc.
There is a myth that if I have only one goat he will be attached to me. This is not the case and should never be the reason for only getting one goat.
Two goats will keep each other company. The goats will be just as attached to you as each other. But when you are not around they will enjoy each others company and can play and jump and just enjoy being goats. The chances of them trying to get out are slimmer (I say slimmer since some goats just love to try to get out just because they can) and this will ensure a better chance for a safe life for the animal.
Another myth is that I dont want to have to feed as much, one goat will be easier. The easier part was already explained away with the fact that the goat will be happier and not crying, escaping etc. The work that is involved with feeding, housing and penning up one goat is the same as it would be for two. As to the feed amount, if you get wethers they hardly eat anything at all (check out more on wethers and why they make great pets).
Some people buy a goat as a companion for a horse. While this sometimes works it is not always the case. A goat needs another of its kind to truly be happy. I have found that having two goats will keep each other company and the horse or pony is just fine with another goat around.
A basic idea for getting a companion goat for a horse is to think of the size relationship. A large breed goat will do better with a horse while a pygmy or nigerian dwarf is not recommended especially alone. But a mini goat breed would do well with a miniature horse or a pony.
Not all horses do well with goats so be sure to introduce them slowly and watch the horses behavior before you leave them alone together. Having a separate area for the goat to retreat to if it needs to is important.