Search code examples
androidkotlinurlrequestretrofit2

403 Error When Making HTTP Request with Retrofit in Kotlin

I’m working on an Android project where I need to send an HTTP request to a search endpoint using the Retrofit library in Kotlin. The request works fine when I enter the URL directly into my browser, but when I try to send the request programmatically, I receive a 403 error.

Here’s the URL I’m trying to request: https://apkpure.com/search?q=free+fire

And this is the code snippet where I set up Retrofit and make the call:

interface ApiService {
  @GET("search?q=free+fire&t=")
  suspend fun searchResults(): Response<String>
}

val retrofit = Retrofit.Builder()
    .baseUrl("https://apkpure.com/")
    .addConverterFactory(ScalarsConverterFactory.create())
    .build()
    .create(ApiService::class.java)

When I execute the request, I get the following error:

Response Error: 403, Error Body: <!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html;

It seems like the server is rejecting my programmatic request but not when I use a web browser. Could this be related to user-agent verification or some other form of request validation that the server is performing?

How can I modify my request to avoid this error and successfully retrieve the data as my browser does?

I tried to add a user agent t my request so this may fix the problem, but it did not, here is the code :

val okHttpClient = OkHttpClient.Builder()
      .addInterceptor { chain ->
        val originalRequest = chain.request()
        val requestWithUserAgent = originalRequest.newBuilder()
          .header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3")
          .build()
        chain.proceed(requestWithUserAgent)
      }
      .build()

    val retrofit = Retrofit.Builder()
      .baseUrl("https://apkpure.com/")
      .client(okHttpClient)
      .addConverterFactory(ScalarsConverterFactory.create())
      .build()
      .create(ApiService::class.java)

Still gives the same error...

Solution

  • After some investigation, I’ve discovered that the website in question relies on JavaScript to process requests. This posed a challenge when using Retrofit for network operations, as Retrofit does not support JavaScript execution. Consequently, the requests sent via Retrofit are treated as if JavaScript is disabled, rendering it incapable of processing web pages that require JavaScript.

    To address this, I experimented with the Scrape{it} library, which successfully handled JavaScript but introduced significant latency. An alternative solution is to utilize a WebView, which allows for JavaScript execution within its environment. By enabling JavaScript in WebView, you can manually extract the results from the page.

    Here’s a snippet to enable JavaScript in WebView:

    WebView myWebView = (WebView) findViewById(R.id.webview);
WebSettings webSettings = myWebView.getSettings();

webSettings.setJavaScriptEnabled(true);

    This approach should provide a more interactive experience, similar to that of a standard browser, allowing for the proper rendering of JavaScript-dependent content.