Its a bit more complicated than that. There are also genes that determine how sensitive various part of your body are to various hormones. A few of these genes get damaged, you can have some odd behaviors. The XY female is an example of that. Its the result of entropy breaking things.
That is...