Django form error when queryset is changed to none or subset

I am working on a dependent dropdown using htmx in Django. Since the dataset is very large I do not want to load all the data in the beginning, and only load the data as required dynamically using htmx. The form working fine when I set the queryset as Model.objects.all() but since it takes a long time to load initially, I changed the queryset to Models.objects.none() or Models.object.all()[:xx] so there is no or minimum data. The form work as expected and the dependent dropdown work nicely, but when I submit the form there is an error saying "Select a valid Choice. That choice is not one of the available choices." My question is does .all() return different objects than .none() or all()[:xx]? If not what cause the error? I have spent hours trying to find the cause of the errors but to no avail. I want to know what cause the error.

Here is the models.py

class Province(models.Model):
    province_code = models.CharField(max_length=2)
    province_name = models.CharField(max_length=30)

class City(models.Model):
    province = models.ForeignKey(Province,on_delete=models.CASCADE)
    city_code = models.CharField(max_length=2)
    city_name = models.CharField(max_length=30)

class Address(models.Model):
    province = models.ForeignKey(Province,on_delete=models.CASCADE)
    city = models.ForeignKey(City, on_delete=models.CASCADE)
    address = models.CharField(max_length=50)

Here is the forms.py

class AddressForm(ModelForm):
    class Meta:
        model = Addreess
        fields ='__all__'
        widgets = {
            "province": form.Select(
                attrs={
                    "hx-get":"import/load-city/",
                    "hx-target":"#city",
                    "hx-trigger":"change",
                    "hx-swap":"innerHTML",
                }
            ),
            "city": form.Select(
                attrs={
                    "id":"city"
               }
            ),
         }

    def __init__(self, *args, **kwargs):
         super(AddressForm, self).__init__(*args,**kwargs)
         self.fields["province"].queryset = Province.objects.all()
         self.fields["city"].queryset = City.objects.all()  
         ## comment: error if .none() or .all()[:xx]

Here is the views.py

class AddressCreateView(CreateView):
     model = Address
     form_class = AddressForm
     template_name = 'address_create.html'
     success_url ='/'

class LoadCityView(View):
     def get(self,request):
         if request.GET.get("province"):
             province = request.GET.get("province")
         else:
             province=None
         cities = City.objects.filter(province=province)
         context = {"cities":cities}
         return render(request,"load_cities.html",context)

Here is the urls.py

urlpatterns = [
    path('address-create/', AddressCreateView.as_view(), name='address-create'),
    path('load-city/', LoadCityView.as_view(), name='load-city'),
]

Here is the template

address_create.html

<!DOCTYPE html>
<html>
    <head>
    </head>
    <body>
        <form method="post">
            {% csrf_token %}
            {{ form.as_p }}
            <input type="submit">
        </form>
        <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.7.1/jquery.min.js"></script>
        <script src="https://unpkg.com/[email protected]"></script>
        <script>
            document.body.addEventListener('htmx:configRequest', (event) =>
             {
                 event.detail.headers['X-CSRFToken'] = '{{csrf_token}}';
             })
        </script>
    </body>
</html>

load_cities.html

<option value="">------------</option>
{% for city in cities %}
<option value="{{city.id}}">{{city.city_code}} {{city.city_name}}</option>
{% endfor %}

Solution

Don't set the queryset, set the .choices to empty, this will prevent rendering the items, but still do the validation properly:

class AddressForm(ModelForm):
    class Meta:
        model = Addreess
        fields = '__all__'
        widgets = {
            'province': form.Select(
                attrs={
                    'hx-get': 'import/load-city/',
                    'hx-target': '#city',
                    'hx-trigger': 'change',
                    'hx-swap': 'innerHTML',
                }
            ),
            'city': form.Select(attrs={'id': 'city'}),
        }

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.fields['city'].choices = []

this will still validate the item with the .queryset that contains all items, and thus will make filtered queries to the database, but omit rendering these choices.

What you are trying to do, lazy loading items with AJAX has been wrapped in a package named django-select2 ^{[readthedocs.io]} it might be better not to reinvent the wheel, and let the package handle this.

Note: Since PEP-3135 ^[pep], you don't need to call super(…) with parameters if the first parameter is the class in which you define the method, and the second is the first parameter (usually self) of the function.