In this chapter, we looked under the hood at the components that make a solid web scraping system. We used colly to scrape HTML pages that did not require JavaScript. We used chrome-protocol to drive web browsers to scrape sites that do require JavaScript. Finally, we examined dataflowkit and saw how its architecture opens the door for building distributed web crawlers. There is more to learn and do when it comes to building distributed systems in Go, but this is where the scope of this book ends. I hope you check out some other publications on building applications in Go and continue to hone your skills!
- Tech Categories
- Best Sellers
- New Releases
- Books
- Videos
- Audiobooks
Tech Categories Popular Audiobooks
- Articles
- Newsletters
- Free Learning
You're reading from Go Web Scraping Quick Start Guide
Vincent Smith has been a software engineer for 10 years, having worked in various fields from health and IT to machine learning, and large-scale web scrapers. He has worked for both large-scale Fortune 500 companies and start-ups alike and has sharpened his skills from the best of both worlds. While obtaining a degree in electrical engineering, he learned the foundations of writing good code through his Java courses. These basics helped spur his career in software development early in his professional career in order to provide support for his team. He fell in love with the process of teaching computers how to behave and set him on the path he still walks today.
Read more about Vincent Smith
Unlock this book and the full library FREE for 7 days
Author (1)
Vincent Smith has been a software engineer for 10 years, having worked in various fields from health and IT to machine learning, and large-scale web scrapers. He has worked for both large-scale Fortune 500 companies and start-ups alike and has sharpened his skills from the best of both worlds. While obtaining a degree in electrical engineering, he learned the foundations of writing good code through his Java courses. These basics helped spur his career in software development early in his professional career in order to provide support for his team. He fell in love with the process of teaching computers how to behave and set him on the path he still walks today.
Read more about Vincent Smith