🔄 Scraping In Progress

Initializing...

0 / 0 sources 0s

Current Source -

Leads Found 0

Errors 0

Shiiman Leads

Home

Dashboard

Sources AI Lead Generator

U

Loading...

Loading...

email@example.com

Menu

U

Loading...

Loading...

Home

Dashboard

Profile

Sources

AI Lead Generator

Logout

My Data Sources

Discovery - Find URLs to Monitor

AI Assist

Describe your niche or target segment, then AI suggests directories, maps, permit portals, and high-intent pages you can save as monitored sources.

Popular blueprints

Plan usage and inbound API tokens

No Sources Yet

Add your first data source to start extracting leads automatically

Screenshot

1 / 1

Add Source

Source Name *

Website URL *

The AI bot will visit this page and extract data automatically

API candidates

Probed with f=json, where=1=1, outFields=* — pick the layer that matches your data (higher row count usually means the main table).

Scraping Method * Choose "AI Website Scraper" for regular websites. Choose "JSON API" if you have a direct API endpoint (like ArcGIS FeatureServer URLs).

📡 JSON API Configuration

Saving any source (including JSON API) requires an active paid plan—same as website scrapers. Your profile (ICP fields) is separate and does not need a subscription.

HTTP Method

Query Parameters (JSON format) For ArcGIS: Add where, outFields, f, resultRecordCount. See ArcGIS Guide

POST body format Check DevTools → Headers → Request content-type. Phoenix uses form, not JSON.

POST body (optional) If empty, Query Parameters JSON is used as the POST body.

Optional request headers (JSON) Rare. Try without first. Load balancers may need a fresh session cookie.

JSONPath Filter (Optional) Extract nested data using JSONPath syntax (leave empty for auto-detect)

🗺️ ArcGIS Hub Setup

Paste the ArcGIS Hub "explore" or "datasets" URL. The system will open it with Playwright, capture the API endpoint + session cookies, and scrape via JSON automatically.

Example: https://data.city.gov/datasets/abcd1234_0/explore

Navigation Instructions (JSON - Optional) Optional: Add click/fill/wait actions before API extraction. Types: "click", "fill", "wait". Max 3 instructions.

Date Field for Year Split (Optional) Use the dataset date column to split large ArcGIS pulls by year (e.g., ApplicationDate).

⚙️ Universal Engine (optional)

Filter and map API fields so only qualified leads are saved. Same logic for any sector (permits, jobs, real estate).

Use Universal Engine for this source

Field mapping (API name → your name) JSON Map API field names to your preferred names. Leave empty to keep raw keys.

Filters (only save leads that pass) JSON Must be a JSON array [...] of rules (not a single {...} unless it auto-wraps). Operators: >, <, ==, contains, in, between, days_ago. Leave empty to save all.

ArcGIS where clause (optional)

🤖 AI Instructions (Describe What You Want)

📅 Need to add dates?

Click a button to insert date into your instructions:

💡 Dates update automatically when you scrape - no need to change them!

✨ Just describe what you want in plain English! AI will figure out how to navigate, find buttons, extract data, and handle multiple pages automatically.

💡 Example Prompts:

"Extract all building permits from the table"
"Click 'Show Table' button, then extract permit data"
"Find commercial permits and extract from all pages"
"Click Building Permits tab, open table, get all data"

What fields do you want to extract? Just list the fields you want, separated by commas. AI will extract them automatically.

Use Residential Proxy When checked, JSON API requests use the first PROXY_URLS entry in server .env (direct connection if unchecked or PROXY_URLS empty).

🎭 Force Playwright Only Prevent fallback to other methods. Always use Playwright browser automation.

Include Keywords (comma-separated)

Exclude Keywords (comma-separated)

Display Columns (Optional) Leave empty for auto-detection. Comma-separated list of columns to show in your dashboard table.

U

User Name

user@example.com

📋 Account Information

Account Status: ✓ Active

Member Since: -

User ID: -

💼 Current Plan

Current Plan

Professional

$299/mo

✓ 10,000 leads/month
✓ All data sources
✓ Advanced filtering
✓ API access
✓ Priority support