IssueRaiser

would you mind sharing some details about how you are scraping?

would you mind sharing some details about how you are scraping? I am also trying to build a crawler. The problem is each site has it's own html structure. How do you handle this? Have you written scraping rules for each site? Which is a nightmare to maintain especially when you have lot of sites to crawl.

Posted by mandeepj on May 14, 2015, 7:06 PM

Tags

Related issues

Raise real-world problems, find people facing the same issue, and discuss solutions.6,500 issues0 comments

hackernews (Other)

would you mind sharing some details about how you are scraping?

mandeepjMay 14, 2015, 7:06 PM

would you mind sharing some details about how you are scraping? I am also trying to build a crawler. The problem is each site has it's own html structure. How do you handle this? Have you written scraping rules for each site? Which is a nightmare to maintain especially when you have lot of sites to crawl.

Comments (0)

No comments yet. Start the discussion.

Comment on this issue