Skip to content
Snowey Labs

8 min read

The lead pipeline: Scrapling to JSON to a Google Sheet

How the script that pulls business names and phones into a spreadsheet actually ran, what broke, and what it cost.

The problem

I wanted a list of local businesses in one industry and one city: name, website, phone, email, address. The end shape is a Google Sheet, because that is what gets shared and worked from. I had done this before with a paid API. This time I wanted it free.

What I used and why not the alternatives

The older way, an earlier directive in the same repo, went through Apify (code_crafter/leads-finder). It works, but it costs money per run, and the cost scales with the number of leads. Scrapling is a Python scraping library that does headless stealth browsing for free. I already had it installed for other work, so I wrote a script against Google Maps directly instead of paying for someone else’s wrapper around it.

I also looked at paginiaurii.ro, a Romanian business directory. It is a static page, so scraping it is cheap: one pull got 1,303 rows. But it lists the SRL legal name, not the trading name, and it has no website field. No website means no email, and email is half the point of the list. I rejected it for that reason alone.

Setup

The install is two commands:

pip install "scrapling[all]>=0.4.15"
scrapling install --force

The second command downloads the browser binaries Scrapling drives. Skip it and there is no browser for the script to drive.

One thing that isn’t obvious: the interpreter matters. On my machine the repo’s own .venv has neither scrapling nor gspread/pandas/dotenv. The one that has both is the system-wide Python 3.12 install, run with PYTHONIOENCODING=utf-8 set, because the default Windows console codepage (cp1252) crashes on Romanian names with diacritics the moment you print one.

The commands

The scrape:

python execution/scrape_leads_scrapling.py --industry "Restaurants" \
    --location "Cluj-Napoca, Romania" --wanted 50

And the offline check that runs the pure logic (deduping, phone/website keys, email filtering) without touching a browser or the network:

python execution/scrape_leads_scrapling.py --self-check

If you’re redirecting the scrape’s output to a log file, add -u (unbuffered) or the prints get stuck in a buffer and the log file looks empty while the script is still running:

python -u execution/scrape_leads_scrapling.py --industry "Restaurants" \
    --location "Cluj-Napoca, Romania" --wanted 50 > run.log 2>&1

The JSON then went into a Google Sheet with the upload script the repo already had (update_sheet.py, unchanged), and I proved the round trip by reading the sheet back with read_sheet.py: 50 rows out, 50 rows back. Done means that number matches, not that the upload command ran.

How it works

Three stages, each narrower than the last:

  1. Feed. Google Maps search results for "<industry> in <location>". A headless stealth browser scrolls the results feed until it has roughly double the wanted count (over-fetching, because some rows won’t survive the later stages) or the feed runs dry.
  2. Place pages. Each result’s Maps detail page gets fetched (three at a time, to stay polite) for its website, phone and address, pulled from the page’s own data-item-id buttons.
  3. Business sites. After the rows are deduplicated, filtered to the requested city and capped at the wanted count, the script visits the website of every remaining place that has one (43 of the 50 in this run): the homepage first and, if that finds nothing, up to two pages whose address looks like a contact page (despre, about, contact, kontakt, impressum), looking for an email address.

Then it writes the JSON. The dedupe key is the website domain or the phone once you strip formatting; the city filter runs before the cap, so a place in the wrong town never takes a slot from one in the right town.

The consent cookie that makes Google Maps show results instead of an EU consent wall:

CONSENT_COOKIE = [{"name": "SOCS", "value": "CAESHAgBEhIaAB", "domain": ".google.com", "path": "/"}]

The phone dedupe key, which collapses 0264-592 022 and +40 264 592 022 into the same lead:

def phone_key(phone):
    digits = re.sub(r"\D", "", phone or "")
    return digits[-9:] if len(digits) >= 9 else ""

Folding diacritics so a city filter matches both spellings of a Romanian place name:

def fold(text):
    stripped = unicodedata.normalize("NFKD", text or "")
    return "".join(c for c in stripped if not unicodedata.combining(c)).lower()

And the junk pattern that throws out addresses that are technically emails but never belong to the business, like the placeholder email a WordPress theme ships with:

EMAIL_JUNK = re.compile(
    r"(sentry|wixpress|godaddy|u003|@2x|"
    r"@(theme|example|domain|yourdomain|yoursite|mysite|email|test|sample|company)\.|"
    r"\.png$|\.jpe?g$|\.gif$|\.webp$|\.svg$)", re.I
)

What broke, in order

Six things, in the order I hit them:

  1. The EU consent wall. Every request came back as a Google consent page instead of search results, until I set the SOCS cookie above.
  2. Cookies on the wrong call. I first passed the consent cookie to the per-page session.fetch() call. It was silently ignored: every fetch still landed on the consent screen, but returned a 200, so nothing looked like an error. Every field came back empty. The fix was to move the cookie onto the AsyncStealthySession constructor instead, which is where the session actually reads it.
  3. Maps spilling into other towns. A search for Cluj-Napoca returned some places in the neighbouring commune of Florești. I added a city filter that folds diacritics before comparing, so "Floresti" and "Florești" both match (or both fail to match "Cluj-Napoca", which is the point).
  4. The console couldn’t print the results. Windows’ default console codepage (cp1252) can’t encode some Romanian characters, so printing a business name crashed the run. Fixed by setting PYTHONIOENCODING=utf-8 before running.
  5. The log file looked empty. Redirecting stdout to a file buffers the output, so tail-ing the log while the script ran showed nothing for minutes. Running with python -u disables that buffering.
  6. A theme’s placeholder email got through. One business site’s WordPress theme ships with hello@theme.com baked into the template, and my email filter didn’t catch it on the first pass. I added theme to the junk-domain list in EMAIL_JUNK and re-applied the filter to the already-scraped JSON, offline, without re-running the browser stages.

Two smaller failures happened during the run itself and didn’t need a code fix: one business site’s hostname did not resolve, and one Maps place page timed out on the first attempt and succeeded on the retry. Both are handled by the script’s existing retry logic.

What it cost

Nothing in API fees. No paid service anywhere in the pipeline: Scrapling is free, the browser it drives is the one it installs itself, and the sheet upload uses a Google account that was already authenticated. The only cost is time: a full run for 50 leads took about six minutes end to end, on one Windows machine, no cloud, no parallel workers beyond the handful of concurrent browser pages inside the script itself.

For that run the feed produced 106 candidate places, 78 of those got a detail page fetched, 43 had a website worth checking for an email, and 50 leads were written after deduping and filtering to the requested city. Fill rates on the 50: 49 had a phone number, 43 had a website, and 25 had an email (the script’s own count during the run said 26; one of those was the WordPress theme placeholder from bug 6 above, caught and removed by the offline re-filter after the run finished).

Where the code lives now

The script that actually ran is execution/scrape_leads_scrapling.py in the working directory it was built in. A cleaned-up, reusable version of the same idea was packaged afterwards as a Claude Code skill (sscraper), with separate scripts for scraping and for pushing rows into a Google Sheet. That packaged version was tested against fixtures, not against a live Google Maps run. The numbers in this guide come from the original script, run live, not from the packaged one. If you use the skill, treat its first real run as the actual test.

Rules I kept

  • Nothing gets sent to anyone. The pipeline collects and writes a spreadsheet; nobody gets emailed, called or contacted by anything in this pipeline.
  • Requests are paced. Detail pages are fetched three at a time, with a delay before each one, so the target sites see a slow, ordinary visitor rather than a burst of traffic.
  • Scraped text is data, not instructions. Nothing pulled from a page, an aria-label or a business’s own site gets treated as a command, no matter what it says.