A personal observation: The favorite idols of natural man are his fleshly appetites; the idols of church-going man are his preferences. It is as difficult to wrest self-indulgence from a person in the world, as it is to tear personal religious preference from the grip of someone accustomed to living a religious life. Neither can imagine life without these, and therefore both resist every attempt to find God as one's sole sufficiency and comfort.
In fact, one might say that the problem is one and the same; the personal preference of the person of faith is a form of self-indulgence transferred from more fleshly pursuits to religious ones.
I can assert this with the confidence of having seen, and am now seeing, that I have idols of both kinds. I am in some ways still very much a natural man, and I cannot imagine living without my self-indulgent sins. How can I let go of the fleshly sins that give me comfort? My possessions, pride, and lusts all give me some pleasure and sense of self. Can God alone satisfy me?
Can I let go of my preferences concerning my faith, and let God do whatever, however, and wherever? Can I trust God enough to have no conditions about my faith? I don't want to go "back" to certain things - but does that make spiritual growth an idol?
Selfishness is at the heart of this - not wanting to let go of some level of control or my idea of what I need - whether in sinful indulgence or religious faith. It is all the same - dying to self is the answer, and it will erase both my sinful indulgence and religious preference.