I would love to have hadoop and few other packages in newer versions that the current ambari with HDP2.6.3 allows.
Is there an option for this kind of single components version upgrades?
This feature will not be ready until Ambari 3.0. See AMBARI-18678 & AMBARI-14714
Depending on what you want to upgrade, though, I wouldn't suggest it.
For example, HBase, Hive and Spark do not yet support Hadoop 3.0. The streaming components of HDP like Spark, Kafka, NiFi seem to release versions more frequently, and there are ways outside of Ambari to upgrade those.
You don't need HDP, or Ambari to manage your Hadoop, but it does make a nice package and central management component for the cluster. If you upgrade pieces on your own, you risk incompatibility.
HDP is tested as a whole unit. The Hortonworks repos that you setup in Ambari limit what component versions that are available to you, but this does not stop you from using your own repositories plus Puppet/Chef from installing additional software into your Hadoop environment. The only thing you lose at that point, is management and configuration from Ambari.
You could try to define your own Ambari Mpacks to install additional software, but make sure you have the resources to maintain it.