I would say that it definitely is difficult for a native speaker of English to learn Polish if that person only comes into contact with the Polish language for the first time as an adult. Of course, if a child is lucky enough to have a Polish Mum and an American Dad, then it's different. I have a British friend who lives in Poland with his Polish wife. They have two daughters who switch from one language to the other without batting an eyelid depending on which parent they are talking to. Since birth, they have had time to get their tongues around Polish pronunciation and a child's mind is a real language sponge.
However, after we become adults, I think something happens in the brain which makes language acquisition much harder. No matter how hard I try, I'm never going to be able to hide my accent when speaking in Polish or get my tongue perfectly around those awful Polish tongue-twisters that Poles find so amusing to hear me attempt to say; I started too late. Some Polish sounds simply do not exist in English. And for a native speaker of English it is often difficult just to differentiate between some sounds let alone pronounce them; I've had people spending ages in the pub trying to get me to hear the difference between "ś" and "sz". And it took about five years of living in Poland before I could hear the difference.
On top of Polish pronunciation difficulties, you've also got Polish grammar. This is a total nightmare for native speakers of English because the grammar system is totally different to theirs. Polish is a Slavic language and as such has a case system. The case system means that nouns and adjectives are declined to provide the meaning that prepositions tend to provide in English. In English, whether it's "on the table", "under the table", or "behind the table", the word "table" always remains the same. Not so in Polish. There are seven different cases and so seven different words for "table" depending on the relation of the object in question to the table. As if that didn't make things hard enough, you also have to decline the adjectives as well so there are seven ways of saying "wooden table". And to make things even more complicated you've got three different genders of nouns which all decline in different ways. Polish grammar is no walk in the park.
Word order is also different from English. In the homework that I marked when I worked as an English teacher in Poland, I would always get sentences like "in Torun" are lots of shops" instead of "There are lots of shops in Toruń". This mistake reflects thinking in Polish. Working as a translator from Polish to English, I often have to put what's at the start of the Polish sentence at the end of the English sentence. Word order differences only add to the difficulties of the student of Polish.
Even if you've started to get your tongue around the pronunciation and your head around the grammar and word order, there is still a wealth of idioms and slang which is often bewildering to the uninitiated.
So all in all, what with Polish pronunciation, grammar, word order, idioms and slang, I would say that yes, it most definitely is difficult for an adult native speaker of English to learn Polish although not impossible. Every journey starts with the first step.