PSA: I was dead wrong about Python being too slow for real work

I spent 6 months avoiding Python for anything beyond scripts because I thought it would be too sluggish for actual projects, but then I built a small web scraper that processed 10,000 pages in under 15 minutes and it changed my mind completely. Has anyone else had a similar moment where a language they dismissed turned out to be way more capable than they thought?

2 comments

2 Comments

keithyoung6d ago

Built a similar scraper a while back and had the same wake up call. The secret sauce for me was just using the right tools like requests with async and BeautifulSoup for parsing. Once I figured out async properly, the speed difference was night and day. Your mileage may vary depending on what you're scraping and how heavy the HTML is, but for standard stuff Python absolutely keeps up.

avery_walker302d ago

Did you run into any issues with rate limiting when you scaled up the async requests? I tried a similar setup but kept getting blocked after like 50 pages because the site caught on to the rapid hits. Had to add random delays and rotate user agents to make it work, which kind of killed the speed advantage I was hoping for. Curious if you found a cleaner way around that or if you just accepted the slower pace for bigger jobs. Async is DEFINITELY the way to go for speed but it feels like sites fight back harder now than they did a few years ago.