Opanlayers current version has layer and source objects to view vector or tile images.
Layer Types: Image, Tile and Vector
Source Types: ImageWMS, TileWMS, Geojson, OSM, Raster, Vector, …
So how can I decide which source Works with which layer? I could not found in openlayers documents page.
The type of source accepted is shown in the documentation for the layer type. It is correct for Image, Vector and VectorTile. Where subclasses are listed in the documentation for the source types they are usually used instead of the base class. The documentation confusingly shows the base class Tile as the source type for Tile layers. Subclasses of that include VectorTile which isn't appropriate as it is only used by VectorTile layers and Tile layers only use subclasses of TileImage or (rarely) TileDebug or UFTGrid. OSM is a subclass of XYZ. There is no GeoJSON source type in OpenLayers 5, it is just one of the formats used by Vector sources.
https://openlayers.org/en/latest/apidoc/module-ol_layer_Tile-TileLayer.html uses subclasses of https://openlayers.org/en/latest/apidoc/module-ol_source_TileImage-TileImage.html or https://openlayers.org/en/latest/apidoc/module-ol_source_TileDebug-TileDebug.html or https://openlayers.org/en/latest/apidoc/module-ol_source_UTFGrid-UTFGrid.html
https://openlayers.org/en/latest/apidoc/module-ol_layer_Image-ImageLayer.html uses subclasses of https://openlayers.org/en/latest/apidoc/module-ol_source_Image-ImageSource.html
https://openlayers.org/en/latest/apidoc/module-ol_layer_Vector-VectorLayer.html uses https://openlayers.org/en/latest/apidoc/module-ol_source_Vector-VectorSource.html
https://openlayers.org/en/latest/apidoc/module-ol_layer_VectorTile-VectorTileLayer.html uses https://openlayers.org/en/latest/apidoc/module-ol_source_VectorTile-VectorTile.html