Photo by mostafa meraji on UnsplashTLDR: Cloudflare’s new Captcha system, Turnstile, has noble intentions but it did not stop all the ‘low and slow’ bots I sent at it. Turnstile is still in beta, so it will be improved.
If you work for Cloudflare, email me and I can share more about the advanced persistent bots I use in this article and how to stop them.
Caveats: This was not an exhaustive test. These are initial findings because Turnstile just came out in late September. Please provide ideas for more exhaustive tests and feel free to call me names in the comments.
Setup
Turnstile is Cloudflare’s new Captcha alternative. I used ‘low and slow bots’ (advanced persistent bots) to see how well Turnstile performed. I did this on my own test site so I wasn’t disrupting any production sites.
I tested Turnstile on 2 major browser automation (bot creation) platforms
Selenium and Puppeteer.
I cloned the turnstile demo worker, added in my site and secret keys, and then sent my bots to try to login.
Turnstile did a good job stopping Selenium so I won’t spend time on that.
Turnstile stopped regular Puppeteer however it had trouble with Puppeteer Stealth.
The Main Test
I ran the following test configurations
10 tests with Puppeteer Stealth Headless and Turnstile Managed
10 tests with Puppeteer Stealth Headless and Turnstile Non-Interactive
10 tests with Puppeteer Stealth Headless and Turnstile Invisible
Turnstile stopped these headless browsers. They seem to be doing a good job picking up on the browser fingerprints left by Puppeteer Stealth.
I then ran
10 tests with Puppeteer Stealth Non-Headless and Turnstile Managed
10 tests with Puppeteer Stealth Non-Headless and Turnstile Non-Interactive
10 tests with Puppeteer Stealth Non-Headless and Turnstile Invisible
Turnstile did not stop any of these.
I even used the same VPN IP addresses over and over again to give Turnstile an edge. It didn’t help.
My hypothesis on why it failed to stop these is because Turnstile is used to seeing headless bots. If they widen the training scope to include non-Headless browsers, Turnstile may start flagging real users. Again just my speculation, but I see this problem with many bot mitigation solutions.
You may think, so what if it doesn’t stop non-headless bots? Most bots are headless. This is true, but advanced persistent bots are evolving. Bot creators are carding or hacking cheap VPSs to launch ‘low and slow’ non-headless bots. They will go through the effort of running them in non-headless mode because their targets are worth it. The mindset of ‘low and slow’ bot creators is much different than DDOS creators. ‘Low and slow’ bots may only need to hit successfully 10–100 times to make it very profitable for its creators.
The Results
After running 30 tests using various VPN IP addresses, the Puppeteer Stealth non-headless bots got through everytime. Remember these are just my initial findings. Further testing is needed, but for now Cloudflare, get @ me. They’ve created a much better Captcha but I think it can be improved even more.