You are here

Bypassing the "testcookie" anti-webscraping protection

ivan's picture
A few days ago, I noticed that ApkTrack (an Android app I maintain) could no longer query one of the websites it usually obtains data from.
The app works mostly through web scraping and once in a while, the target websites set up new countermeasures to prevent bots from accessing their contents (even innocuous bots such as this app). In this post, we'll see how the protection I encountered this week-end was bypassed.

It all began when I noticed that a website (whose identity will not be disclosed) returned the following script in lieu of the expected data:

<html>
 
<body>
    <script type="text/javascript" src="/aes.min.js"></script>
    <script>
        function toNumbers(d) {
            var e = [];
            d.replace(/(..)/g, function(d) {
                e.push(parseInt(d, 16))
            });
            return e
        }
 
        function toHex() {
            for (var d = [], d = 1 == arguments.length && arguments[0].constructor == Array ? arguments[0] : arguments, e = "", f = 0; f < d.length; f++) e += (16 > d[f] ? "0" : "") + d[f].toString(16);
            return e.toLowerCase()
        }
        var a = toNumbers("5d026cff5942d1ab28e3757e4b2e2f87"),
            b = toNumbers("845dd1e672b840c246aa8cfe9b5d3632"),
            c = toNumbers("e48176221e1325e09b9a959370446f05");
        var now = new Date(),
            time = now.getTime();
        time += 3600 * 1000 * 24;
        now.setTime(time);
        document.cookie = "BKS=" + toHex(slowAES.decrypt(c, 2, a, b)) + "; expires=" + now.toUTCString() + "; path=/";
        location.href = "http://site.com/page/?ckattempt=1";
    </script>
</body>
 
</html>

It's plain to see that this script uses a slow AES implementation to generate a cookie required to browse the target website. I notice that the a, b and c variables of the above script change with every try, and while they kind of look like MD5 hashes, none of them can be reversed easily. Time to dig in.
Ideally, I'd like to read the code which generates these values. I'm in luck: a quick search points me to an nginx module called testcookie.

Reading through the 2000-something lines of code is made difficult by the numerous macros coming from nginx, but I understand the following:

  • a and b are the key and initialization vector (respectively) used for the AES-CBC computation ; c is the data to decipher.
  • The latter is generated the following way: c = AES(MD5($testcookie_session + $testcookie_secret)), those two variables being defined in the nginx configuration. More precisely:
    • According to the documentation, testcookie_session can either be the visitor's IP address (i.e. 127.0.0.1), or their IP concatenated with the browser's user-agent (i.e. 127.0.0.1Mozilla/5.0 (X11; Ubuntu; Linux x86_64; [...]). This part is predictable and can be generated easily.
    • testcookie_secret however is an unknown value. It can be fixed, or random (in which case it changes every time the web server is rebooted).

There are basically two ways to bypass this protection. The first way would be to run the javascript code just like a browser would. The second way is to somehow guess what the cookie's value is expected to be. The former implies a lot of overhead in my tiny Android app, so I start looking into the latter.
I need to find out how the testcookie_session is generated on the target website, since it is configuration-dependant. That part is easy: I take another browser, navigate to the website and compare the cookies: they're identical. This means that only the IP address is used Next, I have to guess testcookie_secret's value. We face the following equation:

  • I know a valid cookie just by visiting the website: 64534e58cbc178830089d06de12c00ed.
  • My IP address at the time was 95.130.11.147.
  • We have established that 64534e58cbc178830089d06de12c00ed = MD5("95.130.11.147" + testcookie_secret).

This is a textbook bruteforce situation. I fireup Hashcat:

PS C:\Users\Ivan\oclHashcat-1.33> .\oclHashcat64.exe -m0 .\targets\site.txt -a7 95.130.11.147 .\dicts\wordlist.txt
oclHashcat v1.33 starting...
[...]
64534e58cbc178830089d06de12c00ed:95.130.11.147keepmesecret

The a7 option corresponds to a hybrid attack, which means that every word from the dictionary is prefixed with an arbitrary string (here, my IP address). After a while, Hashcat proudly announces the result: testcookie_secret = keepmesecret.
I actually guessed that value before the bruteforce had ended for a simple reason: keepmesecret is the example value given in the documentation and I had tested it manually. When in doubt, always assume the sysadmin was lazy.

We now have everything needed to forge our cookies, and computing a MD5 hash before each request is all it takes to bypass the protection.

EDIT : Following this post, testcookie_secret's minimum size has been increased to 32 characters in the latest version of the script.

Comments

Hello, I am Theresa Williams After being in relationship with Anderson for years, he broke up with me, I did everything possible to bring him back but all was in vain, I wanted him back so much because of the love I have for him, I begged him with everything, I made promises but he refused. I explained my problem to my friend and she suggested that I should rather contact a spell caster that could help me cast a spell to bring him back but I am the type that never believed in spell, I had no choice than to try it, I mailed the spell caster, and he told me there was no problem that everything will be okay before three days, that my ex will return to me before three days, he cast the spell and surprisingly in the second day, it was around 4 pm. My ex called me, I was so surprised, I answered the call and all he said was that he was so sorry for everything that happened that he wanted me to return to him, that he loves me so much. I was so happy and went to him that was how we started living together happily again. Since then, I have made promise that anybody I know that have a relationship problem, I would be of help to such person by referring him or her to the only real and powerful spell caster who helped me with my own problem. email: drogunduspellcaster@gmail.com you can email him if you need his assistance in your relationship or any other Case.

1) Love Spells
2) Lost Love Spells
3) Divorce Spells
4) Marriage Spells
5) Binding Spell.
6) Breakup Spells
7) Banish a past Lover
8.) You want to be promoted in your office/ Lottery spell
9) want to satisfy your lover
Contact this great man if you are having any problem for a lasting solution
through drogunduspellcaster@gmail.com

Hey guys, I'm so excited of getting my husband back after he left me and our 3 kids for another woman.
After 2 years of marriage, me and my husband has been into one quarrel or the other until he finally left me and moved to California to be with another woman. i felt my life was over and my kids thought they would never see their father again. i tried to be strong just for the kids but i could not control the pains that torments my heart, my heart was filled with sorrows and pains because i was really in love with my husband. Every day and night i think of him and always wish he would come back to me, I was really upset and i needed help, so i searched for help online and I came across a website that suggested that Dr Unity can help get ex back fast. So, I felt I should give him a try. I contacted him and he told me what to do and i did it then he did a (Love spell) for me. 28 hours later, my husband really called me and told me that he miss me and the kids so much, So Amazing!! So that was how he came back that same day,with lots of love and joy,and he apologized for his mistake,and for the pain he caused me and the kids. Then from that day,our Marriage was now stronger than how it were before,All thanks to Dr Unity. he is so powerful and i decided to share my story on the internet that Dr Unity real and powerful spell caster who i will always pray to live long to help his children in the time of trouble, if you are here and you need your Ex back or your husband moved to another woman, do not cry anymore, contact this powerful spell caster now. Here’s his contact: Email him at: Unityspelltemple@gmail.com ,
you can also call him or add him on Whats-app: +2348071622464 ,
his website:http://unityspelltemple.yolasite.com ,
Jessica, 26 years, Texas, USA.

Hey guys, I'm so excited of getting my husband back after he left me and our 3 kids for another woman.
After 2 years of marriage, me and my husband has been into one quarrel or the other until he finally left me and moved to California to be with another woman. i felt my life was over and my kids thought they would never see their father again. i tried to be strong just for the kids but i could not control the pains that torments my heart, my heart was filled with sorrows and pains because i was really in love with my husband. Every day and night i think of him and always wish he would come back to me, I was really upset and i needed help, so i searched for help online and I came across a website that suggested that Dr Unity can help get ex back fast. So, I felt I should give him a try. I contacted him and he told me what to do and i did it then he did a (Love spell) for me. 28 hours later, my husband really called me and told me that he miss me and the kids so much, So Amazing!! So that was how he came back that same day,with lots of love and joy,and he apologized for his mistake,and for the pain he caused me and the kids. Then from that day,our Marriage was now stronger than how it were before,All thanks to Dr Unity. he is so powerful and i decided to share my story on the internet that Dr Unity real and powerful spell caster who i will always pray to live long to help his children in the time of trouble, if you are here and you need your Ex back or your husband moved to another woman, do not cry anymore, contact this powerful spell caster now. Here’s his contact: Email him at: Unityspelltemple@gmail.com ,
you can also call him or add him on Whats-app: +2348071622464 ,
his website:http://unityspelltemple.yolasite.com ,
Jessica, 26 years, Texas, USA.

Please everyone need to read this
Some time things you don't believe can just happen.
My name is Mrs grace tessy am 29 years old i got married
at the age of 26 i have only one child and i was living happily .After three
year of my marriage my husband behavior became so strange and i don't
really understand what was going on, he packed out of the house to another
woman i love him so much that i never dream of losing him, i try my
possible best to make sure that my husband get back to me but all to no
avail i cry and i cry seeking for help i discussed it with my best friend
and she promise to help me of a SPIRITUAL SPELL CASTER called DR BRIGHT,he
is a very great man and a real man that can be trusted and there is nothing
concerning love issues he cannot do that is why they call him the great
doctor. I contacted his email address at (
templeoflightandsolutions@gmail.com) And i told him everything that
happen all he told me is that i should not worry that all my problems will
be solved immediately. He told me what to do to get my husband back and i
did, he said after 3 days my husband will come back to me and start
begging, it really happen i was very surprise and very, very happy our
relationship is very tight and we both live happily again.
So my friends if you have any issues and you need the help of a real spell
caster please try to contact this same email address (
templeoflightandsolutions@gmail.com

HE FIX THESE FOLLOWING PROBLEMS TO ALL
ACROSS THE GLOBE ON:
1. Getting your lover or husband back
2. Spiritual bulletproof
3. Training
4. Money spell
5. Long life spell
6. Prosperity spell
7. Protection spell
8. Get a job spell
9. Becoming a manager spell
10. Get a huge loan without paying any fee spell
11. Getting your scam money back
12. Child spell
13. Pregnancy spell
14. Freedom spell
15. Love spell
16, vanishing spell
17. Invisible human spell
18. Success or pass spell
19. Marriage spell
20. Avenging spell
21. Popularity spell
22. Killing spell
23. Cancer spell ECT
help contact him for help via email: [templeoflightandsolutions@gmail.com] he is a great man THANKS...

This write up is a testimony of a spell caster i met while searching for spell casters who would help me get my ex Husband back to me and i am very happy to say that my ex Husband is back to me and is presently my husband. My name is Janet Rothschild and i am currently married to Jude Rothschild. Jude my Husband and we loved each other so much but my parents was against us seeing each other. We had a misunderstanding which was caused by family feud and we broke up but my love for him didn't leave me because he was my first love. I then contacted Doctor BEN to please help me get him back. I am happy today that my ex Husband is now with me. This is all thanks to Doctor BEN and i want all of you who needs help to seek help from this spell caster and your problem would be solved. Contact his email for help at voodoospelltemple80@gmail.com voodoospelltemple80@yahoo.com or call him on +2348133364240

Thank you for reading my message.

BEST REGARD

Add new comment

(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
To prevent automated spam submissions leave this field empty.