Spider Tips

  1. 检查CSS匹配哪些标签,有两种方法:

    • 在浏览器F12工具的Console选项下,通过document.querySelector("ul");
    • [可视化比1好]通过SelectorGadget进行选择

    有时通过CSS来选择元素(尤其是批量选择元素)比用XPath选择的要准确和全面得多。

References

Scrapy - Overview and Demo (web crawling and scraping) (T_T, youtube, as you know, blocked by GFW)