My website doesn't have a search input but it has hyperlinks to other sites should I worry about XSS(cross site scripting) vulnerability.
You should only link to websites that are secure and trusted. An example of a trusted website is one that uses https or one which is ranked high in Google.
If you link to an untrusted website, then its possible that when your visitors click on the link to the untrusted website, they are taken to a malicious website that serves malware or steals user credentials.