medium

Cursor & Infinite Scroll

The gallery has no page numbers — it feeds itself over a JSON cursor API, and the image URLs are nowhere in the page HTML. Find the endpoint and enumerate it.

▹Recognising a JS-gated gallery: curl the page, find no image URLs
▹Finding the XHR/fetch endpoint behind infinite scroll (Network tab)
▹Cursor pagination — follow next until it's null, don't guess the count
▹Enumerating a JSON API and pulling originals from it

Launch the target →

The goal

There's a gallery at /labs/scrape-02/target/. It has no page numbers — scroll and it just keeps loading more. One of its images carries a flag in its metadata. Pull them all, find it, read it.

The flag looks like BTS{...}. Box at the bottom to confirm.

What you're up against

This one is different from scrape-01. Try the obvious thing first:

curl -s https://bobtheskull.org/labs/scrape-02/target/ | grep -oE '\.jpg'

Nothing. The image URLs aren't in the page at all. This gallery is JS-gated — the HTML ships almost empty, and a script fetches the images at runtime as you scroll. curl doesn't run JavaScript, so it never sees them.

You have two ways in:

Drive a real browser (headless: katana -headless -jc, browsertrix) and let the page's own JavaScript do the fetching for you.
Skip the page and talk to the API directly. Far nicer. Open DevTools → Network, scroll the gallery, and watch what it requests. It's hitting a small JSON endpoint, one call per batch. Reading the data source beats scraping the render every time.

The API is a cursor: each response hands you a batch of image URLs and a pointer to the next batch. Follow the pointer until it runs out. Don't guess how many batches there are — the last one tells you it's the last by handing back a null cursor.

Then it's the same finish as before: the flag is in the EXIF of one original, so pull every image and read the bytes, don't just look at the pictures.

The shape of the answer

Find the endpoint in the Network tab — it's the request that fires on scroll.
Enumerate the cursor. Fetch it, take the URLs, take the next, repeat until next is null. This is loop-until-dry again, but the server is honest this time: the null is a real terminator, not a lie.
Parse JSON, don't regex it. jq or a two-line Python beats grepping braces.
Pull originals + read EXIF exactly as in scrape-01.

Work it yourself. Hints below, one step at a time; full solution at the bottom.

Hints

Stuck? Open them one at a time — each reveals a single step.

▸Hint 1 — the page is empty on purpose

curl the target and grep for .jpg — nothing. The URLs aren't in the HTML;
JavaScript fetches them as you scroll. curl doesn't run JS, so either drive a
headless browser, or go find what the JS is calling.

▸Hint 2 — watch the network

Open DevTools -> Network, filter to Fetch/XHR, and scroll the gallery. One
small request fires per batch:

    /labs/scrape-02/api/0.json

▸Hint 3 — it's a cursor

Each response is { "images": [...], "next": N }. The next value is the cursor
for the following batch. Fetch it, take the images, take the next, repeat.

▸Hint 4 — stop at null

Don't guess how many batches there are. The last response hands back
"next": null — that is your terminator. Loop until you see it.

▸Hint 5 — same finish

The flag is in EXIF again. Download every original the API listed and:

    exiftool out/*.jpg | grep -B1 'BTS{'

Capture the flag

Found it? Paste the flag to verify. Checked in your browser — nothing is sent anywhere.