GooglePoiCrawler/README.md

GooglePoiCrawler

Google Poi Crawler

Reviews crawler

Code : swire_shop_review.py Execute:

python swire_shop_review.py [port] [proxyport]

DB information

swire_store_list(line 103): store shop list reviews_table(line 232): store crawler result, use Function save_js_to_db to save parsing data into database

db_columns = ['author_id','author_page','author_name', 'author_image','author_review_count','review_time','review_content','review_image','store_review_time','store_review'] review_process (line 271): store cralwer status

Crawler list

Function get_next_job (line 98): get shop list, data column need ==shop url== (code called item_url), ==fid==, ==shop rating counts== (code called user_ratings_total)

Use ==fid== as all db key

Page down function

Function get_reviews (line 205): Check if the store has reviews first and use shop rating countsdivided by three as page down counts.