Search code examples
pythonapitensorflowparsingdocstring

How to Set up Tensorflow Parser from github source code to Parse Docstrings on there

I wanted to parse the source code on the github of Tensorflow which is located here https://github.com/tensorflow/docs/blob/master/tools/tensorflow_docs/api_generator/parser.py i wanted to parse the docstrings for the API's they have on their website. A sample API can be found here on their website https://www.tensorflow.org/api_docs/python/tf/AggregationMethod and the corresponding code on github too can be found https://github.com/tensorflow/tensorflow/blob/v2.7.0/tensorflow/python/ops/gradients_util.py#L892-L921. This is what i have been trying so far and i have been getting empty results back, my suspicion at first was that i wasn't using the right path for the documents but i tried changing it up as much as i can but i had no luck.

from tensorflow_docs.api_generator import parser

Method signature to_json_file(self, filepath)

print("I am gonna print")

print(parser.get_obj_type("tensorflow_estimator.python.estimator")) print(parser._get_raw_docstring("tensorflow_estimator.python.estimator.estimator"))

print(parser.get_obj_type("_MarkReachedOps")) print(parser._get_raw_docstring("_MarkReachedOps"))

print(parser.get_obj_type("Aggregation_Method")) print(parser._get_raw_docstring("Aggregation_Method"))

print(parser.get_obj_type("tf.estimator.python.estimator.estimator")) print(parser._get_raw_docstring("tf.estimator.python.estimator.estimatorv"))

print(parser.get_obj_type("ForwardAccumulator")) print(parser._get_raw_docstring("ForwardAccumulator"))

print(parser.get_obj_type("autograph.to_code")) print(parser._get_raw_docstring("autograph.to_code"))

#print(parser.to_json_file(tf.autograph.to_code, hello.json ))

print("I am done printing")

Any help will be much appriciated!

Solution 
  • pip install -U tensorflow 

git clone [https://github.com/tensorflow/docs](https://github.com/tensorflow/docs)

pip install docs/

git clone [https://github.com/tensorflow/tensorflow/](https://github.com/tensorflow/tensorflow/)

python tensorflow/tensorflow/tools/docs/generate2.py --output_dir=./here_is_the_docs

    Then the documentation in markdown should be in the here_is_the_docs folder. Please note that the last line will take about 10 minutes to finish running so just be patient with it.