How to convert a dropdown selection menu element in HTML/URL to Pandas Dataframe?

Question

While creating datasets for matching and extracting IDs and SubIDs with their names I have the following code in HTML after getting the file from requests module -

    <div class="feature">
            <h5>Network</h5>
            <div>
                <div class="row">
                    <ul class="tree network-tree">
                    
                        
    <li class="classification class-C  ">
        <span>
        <input type="checkbox" >
        <a href="/network/nt06410+N01032+N01031" target="_blank">nt06410</a> Calcium signaling

        </span>
        <ul>
    <li class="entry class-D network ">
        <span>
        <input type="checkbox" >
        <a href="/entry/N01032" data-entry="N01032" target="_blank">N01032</a>
        
        Mutation-inactivated PRKN to mGluR1 signaling pathway

        </span>
    </li>

    <li class="entry class-D network ">
        <span>
        <input type="checkbox" >
        <a href="/entry/N01031" data-entry="N01031" target="_blank">N01031</a>
        
        Mutation-caused aberrant SNCA to VGCC-Ca2+ -apoptotic pathway

        </span>
    </li>

        </ul>
    </li>

What I want to do is get this particular dropdown selection menu that highlights particular linkages into a pandas dataframe -

ID	Name	Subname	SubID	Network
nt06410	Calcium signaling	Mutation-inactivated PRKN to mGluR1 signaling pathway	N01032	nt06410+N01032+N01031

My code so far has been -

data = soup.find_all("ul", {"class": "tree network-tree"})

# get all list elements
lis = data[0].find_all('li')

# add a helper lambda, just for readability
find_ul = lambda x: x.find_all('ul')
uls = [find_ul(elem) for elem in lis if find_ul(elem) != []]

# use a nested list comprehension to iterate over the <ul> tags
# and extract text from each <li> into sublists
text = [[li.text.encode('utf-8') for li in ul[0].find_all('li')] for ul in uls]

print(text[0][1])

Answer 1

You can use a nested for loop and append items to a list. You will be repeating items in the outer loop e.g. ID for each instance within the inner loop e.g. subID Convert the list of lists to a DataFrame with pandas at the end.

results = []

for network in soup.select('.row .classification'):
    a = network.select_one('a')
    _id = a.text
    _network = a['href'].split('network/')[-1]
    name = a.next_sibling.strip()

    for pathway in network.select('.network'):
        b = pathway.select_one('a')
        subname = b.next_sibling.strip()
        subid = b.text
        results.append([_id, name, subname, subid, _network])
df = pd.DataFrame(results, columns = ['ID', 'Name', 'Subname', 'SubID', 'Network'])

Tested with a related link: https://www.kegg.jp/pathway/hsa05022

N.B. KEGG does offer free JSON downloads of hierarchies.