BobTheSkull logoBobTheSkull
medium

The Defender's View

You just scraped the scrape-01 gallery. Here's the server's access log of it. Find the scraper from the defender's side — the attacker view and defender view of the same event.

  • ▹Spotting enumeration in access logs: sequential paths, volume, cadence
  • ▹Separating a scraper from bots and real users
  • ▹Pivoting from a request pattern to the offender's User-Agent
  • ▹The same instinct loghound/netsentinel automate
Launch the target →

The goal

Flip sides. You've been the one scraping — now you're the one reading the logs.

At /labs/log-01/access.log is a real-shaped nginx access log from the server that hosts scrape-01. Buried in ordinary traffic, one client ran the exact scrape you just did: walked the gallery pages in order and pulled every image. Find them from the defender's side. The flag is in how they announced themselves.

The flag looks like BTS{...}. Box at the bottom to confirm.

Grab the log:

curl -sO https://bobtheskull.org/labs/log-01/access.log

What you're up against

225 lines. Most of it is noise: Googlebot, Bingbot, a few real browsers hitting the homepage, /projects, /favicon.ico, /sitemap.xml. Normal. Boring. Good.

Somewhere in there is one client doing something no human and no search bot ever does: requesting /labs/scrape-01/target/page/1/, then /2/, then /3/ … all the way up, in order, one after another, a fraction of a second apart — and then fetching every tile-*_1600.jpg in a burst.

That shape — sequential paths, high volume, tight cadence, one source — is what enumeration looks like from the server. It's exactly what loghound and netsentinel are built to flag automatically. Here you do it by hand once, so you know what the tools are actually looking at.

The shape of the answer

  • Scope to the interesting paths. grep '/labs/scrape-01/' cuts 225 lines to the handful that matter.
  • Find the one source. One IP owns nearly all of them. Everyone else is scattered. awk '{print $1}' | sort | uniq -c | sort -rn ranks the talkers.
  • Confirm the pattern, don't just trust the count. Are the pages in order? Is the timing inhumanly regular? That's the difference between "popular" and "scraped".
  • Pivot to who they are. Once you have the offender's IP, look at what they sent in their User-Agent. Scrapers are chatty; this one signed its work.

Work it yourself. Hints below, one step at a time.

Hints

Stuck? Open them one at a time — each reveals a single step.

▸Hint 1 — scope to what matters
225 lines is mostly noise. The only requests that matter are the ones
against the scrape-01 target:

    grep '/labs/scrape-01/' access.log
▸Hint 2 — find the one talker
Almost all of those come from a single source. Rank the IPs:

    grep '/labs/scrape-01/' access.log | awk '{print $1}' | sort | uniq -c | sort -rn
▸Hint 3 — is it actually a scrape?
One IP dominates. Confirm the shape: the page numbers climb in order and
the timestamps are a fraction of a second apart. Humans and search bots
don't do that. That's your offender.
▸Hint 4 — who are they?
Pivot from the pattern to the identity. Look at what that IP sent as its
User-Agent (the last quoted field). It signed its work.
Capture the flag

Found it? Paste the flag to verify. Checked in your browser — nothing is sent anywhere.