Rate Limits & Backoff

The goal

The gallery at https://lab.bobtheskull.org/page/1/ works like the others — paginated, an image on the last page carries a flag in its EXIF. But this one fights back: it's rate-limited. Go too fast and it stops answering.

Pull the whole set the polite way, and read the flag. The flag looks like BTS{...}.

What you're up against

Try the obvious hammer — a tight loop over the pages — and watch it fall apart:

for n in $(seq 1 10); do curl -s -o /dev/null -w "%{http_code}\n" \
  https://lab.bobtheskull.org/page/$n/; done

You'll see a couple of 200s, then a wall of 429 Too Many Requests. Every 429 is a page you didn't get. Miss the last page and you never see the flagged image at all.

The server is also telling you exactly how to behave. Look at the response headers:

curl -sI https://lab.bobtheskull.org/page/1/ | grep -i retry-after

Retry-After: 2. That's not decoration — it's the contract. A well-behaved client reads it and waits. This is the whole lab: the fast way fails, the honest way works.

The shape of the answer

• Throttle under the limit. The server allows a small burst then ~2 requests a second. A sleep 0.6 between requests keeps you comfortably inside it.
• Respect Retry-After on a 429. If you do trip it, don't just retry immediately — back off for the number of seconds it told you, then continue. This is the difference between backing off and getting banned.
• Same finish. Once you've pulled every page politely, collect the originals, download them (still throttled — aria2c -j 1 or wget --wait), and read EXIF.

The lesson generalises: an unthrottled loop against someone else's box is indistinguishable from an attack in their logs (see log-01). Rate limits and Retry-After are how servers ask you to be a good citizen — and reading them is the difference between a scraper that keeps working and one that gets blocked.

Work it yourself. Then check the flag.