Web crawling can be easier than you think. Here are some pointers to look for to make your work a lot smoother.