Imagine I give you the following Centaline API. Would it be easier than scraping the Centaline Web by locating each html element and looping through all pages?
So, how to do that?
- Find the APK of your target in Google
- Download and run the script in Github. This script modifies the
network_security_config.xmlfile to allow third parities to listen to network requests.
- Download mitmproxy, a famous man-in-the-middle attach tool.
- Set up the port forwarding in your phone to forward traffic to port 8080
- Download the mitmproxy User Certificate by going to mitm.it in your phone
- Browse your target app and capture traffic in mitmproxy
For an introduction on how to use mitmproxy in Andorid phone, please refer to this Medium post.
For how to modify apk and showing scraping centaline app in action, please refer to my video in LinkedIn.
SSL Pinning will prevent such attacks. But there are ways to bypass SSL pinning. Luciky, all major property listing apps in Hong Kong do not use SSL pinning. haha. For mainland app, they use it. I am still learning how to bypass SSL pinning so that I can scrape some mainland listing apps as well. Will keep you guys updated.
Btw, first post in this discourse channel! I am John