Flutter web scraping using http and html packages not populating any results

Question

I am trying to perform web scraping in a Flutter app using the http and html packages. Specifically, I want to scrape the search results page of Amazon to retrieve information about iPhone listings. However, I'm encountering an issue where the code doesn't populate any results in the app.

Here's the relevant code snippet:

import 'package:flutter/material.dart';
import 'package:http/http.dart' as http;
import 'package:html/parser.dart' as parser;
import 'package:html/dom.dart' as dom;

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Fruit Search',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: SearchPage(),
    );
  }
}

class Article {
  final String title;
  final String url;
  final String urlImage;

  Article({
    required this.title,
    required this.url,
    required this.urlImage,
  });
}

class SearchPage extends StatefulWidget {
  @override
  _SearchPageState createState() => _SearchPageState();
}

class _SearchPageState extends State<SearchPage> {
  List<Article> articles = [];

  @override
  void initState() {
    super.initState();
    getWebsiteData();
  }

  Future<void> getWebsiteData() async {
    final url = Uri.parse('https://www.amazon.com/s?k=iphone');
    final response = await http.get(url);
    dom.Document document = parser.parse(response.body);

    final titles = document
        .querySelectorAll('h2 > a > span')
        .map((element) => element.innerHtml.trim())
        .toList();

    final urls = document
        .querySelectorAll('h2 > a')
        .map(
            (element) => 'https://www.amazon.com/${element.attributes['href']}')
        .toList();

    final urlImage = document
        .querySelectorAll('span > a > div > img')
        .map((element) => element.attributes['src']!)
        .toList();

    setState(() {
      articles = List.generate(
        titles.length,
        (index) => Article(
          title: titles[index],
          url: urls[index],
          urlImage: urlImage[index],
        ),
      );
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Web Scraping')),
      body: ListView.builder(
        itemCount: articles.length,
        padding: const EdgeInsets.all(12),
        itemBuilder: (context, index) {
          final article = articles[index];

          return ListTile(
            leading: Image.network(article.urlImage,
                width: 50, fit: BoxFit.fitHeight),
            title: Text(article.title),
            subtitle: Text(article.url),
          );
        },
      ),
    );
  }
}

I have made sure to include the necessary dependencies in my pubspec.yaml file:

dependencies: http: ^1.1.0 html: ^0.15.4

dditionally, I have set the minSdkVersion to 19 in the Android manifest, although I'm not sure if it's relevant to the issue.

The code runs without any errors, but the ListView.builder in the SearchPage widget doesn't display any results. I have verified that the HTTP request is successful, and I can see the HTML response in the network logs.

I suspect there might be an issue with the CSS selectors used to extract the necessary information from the HTML response. I have inspected the Amazon search results page manually, and it seems that the CSS selectors used in the code should be correct. However, I might be missing something.

I would appreciate any insights or suggestions on what might be causing the issue and how to fix it. Thank you in advance!

Answer 1

Try to run your project with this command flutter run -d chrome --web-browser-flag "--disable-web-security" or add this configuration to your vscode launch.json file

        "name": "xxx",
        "request": "launch",
        "type": "dart",
        "flutterMode": "debug",
        "args": [
            "--web-browser-flag",
            "--disable-web-security"
        ]