If you think working for a contractor is a cherry job, you are twisted in your mind.
ULP said it right. If you don't understand it, we can't explain it. Whether it is a permanent job, or a subcontractor job, or a contractor job, they offered you a position and you turned it down. THAT is their first, and most meaningful impression, of you.
It's a matter of supply and demand. If there was a large supply jobs and there was a demand for employees, you would have been okay to turn down the "offer." But when jobs are scarce, and there are a surplus of people available for the job, don't expect an employer to give you a second chance at a bite at the cherry.
Look at this way: Steve Jobs got a second bite at the cherry with Apple. Although he started the company, it got big, the Board got ugly. He left and they parted on bad terms. Apple was losing. They asked Jobs to come back. He got a second bite at the cherry.
A generic job applicant no. 1 turns down a job offer. The guy behind him in line gets offered and takes the job. It is extremely unlikely that the company will offer another job to generic job applicant no. 1. He got a chance at a bite at the cherry and he declined it. It's the same concept of quitting a job. It is uncommon for a company to not rehire the worker.