GridView OnClick Listener Doesn't Work Sometimes

Question

I've a strange problem. I've a gridView with 9 items (0-8) (can be more/less). As the below picture shows, it shows first 4 items on my device's screen.

A green tick mark is added whenever an item is clicked. It works fine almost every-time. But whenever I scroll down and click quickly, the 5th (Item 4)item's click listener doesn't trigger immediately. It may be a small issue but for user's perspective, sometimes it may be a little annoying. Please tell me how can I make it flawless ? Here is the code of my customAdapter :

 @Override
    public View getView(final int position, View convertView, final ViewGroup parent) {

        View rowView = convertView;
        Holder holder = null;

        if (rowView == null) {
            inflater = (LayoutInflater) mContext.getSystemService(Context.LAYOUT_INFLATER_SERVICE);
            rowView = inflater.inflate(R.layout.single_griditem_gitfamount, null);
            holder = new Holder();
            holder.textViewGiftAmount = (TextView) rowView.findViewById(R.id.textViewGridAmount);
            holder.imageViewGiftTick = (ImageView) rowView.findViewById(R.id.imageViewGridTick);
            holder.cardViewGridAmount = (CardView) rowView.findViewById(R.id.cardViewGridAmount);
            rowView.setTag(holder);
        } else {
            rowView = convertView;
            holder = (Holder) rowView.getTag();
        }
        holder.textViewGiftAmount.setText("Item " + position);

            if (position == clickedpos) {
                holder.imageViewGiftTick.setVisibility(View.VISIBLE);
            }
            else{
                holder.imageViewGiftTick.setVisibility(View.GONE);
            }

         //  final Holder h = holder;
           holder.cardViewGridAmount.setOnClickListener(new View.OnClickListener() {

            @Override
            public void onClick(View v) {
                clickedpos = position;
                notifyDataSetChanged();

            }
        });

        return rowView;
    }

Answer 1

If you are clicking on complete Item then you can choose itemclicklistener

gridview.setOnItemClickListener(new OnItemClickListener() {
        @Override
        public void onItemClick(AdapterView<?> parent, View v, int position, long id) {
            Toast.makeText(MainActivity.this, "" + position, Toast.LENGTH_SHORT).show();
        }
    });

and you can use recyclerview as well.