google-maps-scraper

Savant verdict: Review advised

Security analysis or source-hub review identified patterns requiring attention.

Live evaluation

Not evaluated yet. Workspaces can request a live evaluation.

Security assessment

Scan pending.

Structure

• 2 referenced files aren't in the packagescripts/scrape.py, scripts/scrape.sh. They may be binary, too large to catalog, or missing upstream.
• No license declaredConfirm you may reuse this skill before importing it into your repository.
• The source hub's security checks flagged this skillGen Agent Trust Hub: fail (The skill is a local utility for scraping Google Maps data via a REST API hosted on the user's machine. While it uses shell commands to interact with the local service, no malicious patterns or security risks were identified beyond the inherent risk of processing untrusted web data. The automated alert regarding remote code execution was evaluated as a false positive because it targets localhost.); Snyk: warn (Risk: MEDIUM · 1 issue)

SKILL.md

---
name: google-maps-scraper
description: Google Maps scraper for lead generation. Turns a business type and a place into a clean CSV of local businesses by driving the gosom/google-maps-scraper engine, running locally in Docker as a REST API on localhost:8080. Returns 8 lead fields by default (name, phone, email, website, category, address, rating and review count), plus optional Instagram, Facebook and LinkedIn profiles or all ~34 raw fields. Use when the user wants a lead list, "a list of [businesses] in [place]", local business data, contact enrichment for places (emails, phones, socials), or a whole-city run past Google's ~120 results per search. Do not use for scraping Instagram, TikTok, YouTube, X or any other social media platform directly.
---

# Google Maps Scraper

Turn a business type and a location into clean lead rows by driving the local engine's job API.

## Mental model
The engine (gosom/google-maps-scraper) runs as a local Docker container with a REST API at
`http://localhost:8080`. It has no auth, so it stays bound to localhost. A scrape is an **async job**:
create it, poll until it is done, then download a CSV. One job can run many keywords. Each result has
up to **34 raw fields**.

## Step 0: Check the engine is running
```bash
curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN
```
If `DOWN`: run `docker compose up -d` from the kit root, wait ~10s, and retry. If Docker is not
installed, point the user to `SETUP.md`.

## Step 1: Create a job (`POST /api/v1/jobs`)
**Required fields. The API returns `422` without them:**
- `keywords`: array of search strings. **Bake the location into each term**: `"plumbers in Denver CO"`.
- `lat`, `lon`: **strings**, the city's coordinates: `"39.7392"`, `"-104.9903"`.
- `max_time`: integer **seconds**, the job's wall-clock limit, e.g. `300`. The API stores it as
  nanoseconds internally, but you always send seconds.

**Recommended fields:**
- `depth` (default 10): how far to scroll, so roughly how many listings per keyword. Start at `5`.
- `lang` `"en"`, `zoom` `15` (city level), `radius` `10000` (meters), `fast_mode` `false`.
- **`email` `true`: on by default in the kit.** Emails are the most valuable lead field. The engine
  visits each business website to find them, which is a little slower. Set `false` only for a
  deliberately fast run without emails.

```bash
curl -s -X POST http://localhost:8080/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"name":"job","keywords":["coffee shops in Austin TX"],"lang":"en","zoom":15,
       "lat":"30.2672","lon":"-97.7431","fast_mode":false,"radius":10000,
       "depth":5,"email":true,"max_time":300}'
# → {"id":"<uuid>"}   (HTTP 201; note: lowercase "id")
```

> **Handle two things on every scrape:**
> 1. **Emails: on by default** (`email:true`). Turn them off only if the user wants a fast run.
> 2. **Socials: ask first.** Before creating the job, ask the user once whether they also want
>    Instagram, Facebook and LinkedIn profiles (see "Social profiles" below). Never skip the question silently.

## Step 2: Poll until done (`GET /api/v1/jobs/{id}`)
The response field is `"Status"` (capital S). It moves from `working` to `ok` (success) or `failed`.

> **Harness rule:** the Bash tool **blocks foreground `sleep`**. Run the poll loop as a
> **background** Bash command (`run_in_background: true`) and read its output file when notified.
> Never poll with a foreground `sleep`.

Background poll snippet. It parses the value safely: match up to the closing quote, do not anchor on `$`.
```bash
ID="<uuid>"
for i in $(seq 1 40); do
  S=$(curl -s "http://localhost:8080/api/v1/jobs/$ID" | grep -oE '"Status":"[^"]*"' | head -1 | cut -d'"' -f4)
  echo "status=$S"
  [ "$S" = ok ] && { echo DONE; break; }
  [ "$S" = failed ] && { echo FAILED; break; }
  sleep 15
done
```

## Step 3: Download and parse (`GET /api/v1/jobs/{id}/download`)
```bash
curl -s "http://localhost:8080/api/v1/jobs/$ID/download" -o results.csv
```
CSV columns (34): `input_id, link, title, category, address, open_hours, popular_times, website, phone,
plus_code, review_count, review_rating, reviews_per_rating, latitude, longitude, cid, status,
descriptions, reviews_link, thumbnail, timezone, price_range, data_id, place_id, images, reservations,
order_online, menu, owner, complete_address, about, user_reviews, user_reviews_extended, emails`.

### Return only the 8 lead fields (default)
The raw CSV has 34 columns and most are noise for outreach. **By default, return only the 8 lead
fields**, the data used to contact and qualify a lead, and **drop everything else**:

> `title` (name), `phone`, `emails`, `website`, `category`, `address`, `review_rating`, `review_count`

**Drop by default.** Show these only if the user explicitly asks: `latitude`/`longitude` (no use for
outreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `popular_times`,
`reviews_per_rating`, `reviews_link`, `thumbnail`, `images`, `timezone`, `price_range`, `status`,
`input_id`, `complete_address`, `reservations`, `order_online`, `menu`, `owner`, `about`, `descriptions`,
`user_reviews`, `user_reviews_extended`.

`scripts/scrape.py` already returns exactly the 8 lead fields. Use `--full` to keep all columns, or
`--fields "a,b,c"` to choose your own. It **saves a CSV file by default** (`results-<id>.csv`); pass
`--json` for JSON. If you call the API directly, **strip to the lead fields yourself** before
presenting. Never dump the full 34-column row at the user.

### Social profiles: always ask the user (Instagram, Facebook, LinkedIn)
Google Maps has no social links, so socials are an enrichment step: visit each business `website` and
regex out its Instagram, Facebook and LinkedIn URLs. **Before scraping, ask the user once** whether they
want socials too, unless they already said. If yes, **use the script:** `python3 scripts/scrape.py … --socials`.

- **Token cost. Tell the user this when they ask for socials:** the extraction itself costs **0 LLM
  tokens**, because it is plain HTTP and regex in the script. It adds **~40–50 tokens per business** to
  *your* context only **if** you load the rows into chat, e.g. ~+2k tokens for 50 leads. That is
  negligible if you keep the file on disk and show a sample. Save the file and show a few rows.
- **Do not** fetch each website yourself with WebFetch to find socials. That reads every page into your
  context and costs **thousands of tokens**. The script does it for free, so always prefer `--socials`.
- Socials are **opt-in and slower** (one HTTP fetch per business), and coverage is partial (~40–70%).
  Only businesses that link their socials on their site have them, and no website means no socials.
  Say so if the user expects 100%.

**Shortcuts. Prefer these for common cases:**
- One keyword: `scripts/scrape.sh "<keyword>" <lat> <lon> [depth]` (bash) creates, polls and downloads in one go.
- Auto-geocode (no coordinates): `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]`
  resolves lat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coordinates.
  **Emails come back by default** (`--no-email` skips them). Add `--socials` if the user asked for socials.
- **Batch (many keywords, one job):** `python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"`.
  The API takes a `keywords` array, so put all terms in a single job rather than firing many jobs.
Use the manual curl flow only for custom job bodies, e.g. to set `proxies`.

**Whole-city runs:** when the user says all, every or entire city, or a run returns about 120 rows
(Google's per-search cap), use `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3`.
It splits the city into 3 x 3 tiles (9 jobs) and dedupes on `place_id`. Tiles run one at a time, so a
grid is slower and more likely to trip rate limits: warn once, don't block.

**Deep email pass:** on by default whenever emails are on. Rows with a website but no email get their
contact and about pages checked, and the script prints the found rate. `--no-deep-emails` skips it.

## Other endpoints
- `GET /api/v1/jobs` lists jobs. `DELETE /api/v1/jobs/{id}` deletes a job and frees disk.
- Browser UI and OpenAPI docs: `http://localhost:8080` and `http://localhost:8080/api/docs`.

## Best practices (from the upstream docs)
- **Depth:** higher `depth` returns more results but is slower and more likely to be blocked. Start at 5
  and raise it as needed.
- **One job at a time** locally. Many concurrent jobs without proxies lead to throttling by Google.
- **Email extraction (`email:true`)** visits each business website, so it is slower. It stays **on by
  default** because emails are the key lead field. Pass `--no-email` (script) or `email:false` (raw API)
  for a fast run.
- **`fast_mode:true`** returns reduced data, up to ~21 results per query, faster. Use it for quick lookups.
- **Proxies:** for large or repeated jobs, set `"proxies"` (an array of `socks5://`, `http://` or
  `https://` URLs, auth supported). The engine rotates them. This is the main defense against rate limits.
- **`zoom` and `radius`** set the search area around `lat`/`lon`. Widen `radius` if results are too few.
- **Extended reviews** are available but heavy. Enable them only if the user asks for review text.

## Rate limits, blocks and proxies (read before large jobs)
The engine sends real requests to Google Maps. The upstream project's only formal note is a disclaimer:
> "Please use this scraper responsibly and in accordance with applicable laws and regulations. Unauthorized scraping may violate terms of service."

There is **no published hard threshold** for blocks, so stay conservative:
- Google may **temporarily rate-limit your IP** if you scrape too fast or too much. It clears in
  minutes to hours and does **not** affect the user's Google account, but jobs fail in the meantime.
- **Block signals:** jobs returning `failed`, empty or unusually short results, or a sudden drop in
  row count against an identical earlier run. When you see them, **back off** (pause, lower depth) or add proxies.
- **Concurrency and blocking** (upstream): *"Higher concurrency … can increase blocking or failures,
  especially without proxies. Start with the default for a first run."* Reference throughput is about
  **120 places/min** at `-c 8 -depth 1`. Locally, run **one job at a time** and start at `depth 5`.

**When to add proxies** (upstream: *"For larger scraping jobs, proxies help avoid rate limiting"*):
large jobs, many keywords, repeated or scheduled runs, or after you see block signals.
- Set the `"proxies"` array in the job body. Types: `socks5`, `socks5h`, `http`, `https`.
- Format: `protocol://user:pass@host:port` (auth optional), e.g.
  `"proxies": ["socks5://user:pass@host:port", "http://host2:port2"]`. The engine rotates them automatically.

## Safety and guardrails
- **Warn, don't block.** When a request is large (high `depth`, many keywords, repeated runs, a grid, or
  `email:true`), **proceed with it**, but first print ONE short warning about temporary IP rate limits and
  suggest proxies. Do **not** gate, force-stop or demand confirmation because a normal scrape is big.
  Refuse outright **only** for clearly abusive or illegal use (surveilling individuals, spam, harassment).
- **Never expose** the API beyond `localhost` without an auth proxy. Never print an API key if one exists.
- **Personal data:** scraped emails and phones are personal data. If the user will store or contact them,
  remind them to follow GDPR, CCPA and CAN-SPAM (lawful basis, opt-outs, suppression). Do not help with
  spam or harassment.
- **Google's Terms of Service:** scraping Google Maps is against them. Keep volume modest, treat output as
  leads to verify, and do not resell raw Google data. Refuse uses aimed at surveilling individuals.
- **Output hygiene:** do not dump large CSVs into chat. Save the file to disk, summarize the counts and
  show a few sample rows. Dedupe by `place_id`/`cid` before storing.
- **Disk:** results pile up in the Docker volume. Offer to `DELETE` old jobs from time to time.

## Troubleshooting
- `422 missing max time`: add `max_time` (seconds). `422 missing geo coordinates`: add string `lat`/`lon`.
- Stuck on `working`: lower `depth`, raise `max_time`, or the IP is throttled (add proxies or wait).
- Empty CSV: the keyword is too narrow or the coordinates are wrong. Widen `radius` or fix the coordinates.
- Connection refused: the container is down. Run `docker compose up -d`.