When attempting to merge multiple dataframes, how to resolve "ValueError: If using all scalar values, you must pass an index"

Question

I am trying to fetch and store all historical 1 minute candle data from Bitfinex exchange. When attempting to append new dataframes to an existing one, I get this error "ValueError: If using all scalar values, you must pass an index", despite passing an index in constructor.

Have tried the solution here - to pass an index in the DataFrame constructor: Constructing pandas DataFrame from values in variables gives "ValueError: If using all scalar values, you must pass an index". Its probably something very simple, but have had no luck.

# Example: https://api-pub.bitfinex.com/v2/candles/trade:1m:tBTCUSD/hist?limit=100&start=1549086300000&end=1549174500000
# Params: timeframe, ticker, number of candles, MS start, MS end
# Note: parameter "end" seems to be unnecessary.
# JSON: [[MTS, OPEN, CLOSE, HIGH, LOW, VOLUME],]

import json
import time
import datetime
import requests
import pandas as pd

url = 'https://api-pub.bitfinex.com/v2/'

# Return dataframe of all historical 1m candles

def get_candles_all(symbol):
    symbol = symbol
    limit = 5000
    tf = '1m'
    targettime = (time.time() - 120) * 1000
    start = get_genesis_timestamp(symbol)
    df = get_candles_period('1m', symbol, limit, start)
    while df.index[-1] <= targettime:
        start = df.index[-1] # reset start to last timestamp
        newdata = pd.DataFrame(get_candles_period('1m', symbol, limit, start), index=[0])
        result = df.append(newdata)
        df = result 
    return df


# Return timestamp-indexed dataframe of requested timeframe candles

def get_candles_period(tf, symbol, limit, start):
    symbol = symbol
    response = requests.get(url +"candles/trade:" + tf + ':t' + symbol + '/hist?limit=' + str(limit) + '&start=' + str(start) + '&sort=1').json()
    df = pd.DataFrame(response)
    df.columns = ["MS", "Open", "High", "Low", "Close", "Vol"]
    df.set_index("MS", inplace=True) 
return df

# Return timestamp of first available 1 min candle of given asset

def get_genesis_timestamp(symbol):
    symbol = symbol
    response = requests.get(url + "candles/trade:1m:t" + symbol + '/hist?limit=1&sort=1').json()
    df = pd.DataFrame(response)
    df.columns = ["MS", "Open", "High", "Low", "Close", "Vol"]
    df.set_index("MS", inplace=True) 
    timestamp = df.index[0]
return timestamp

symbol = "ETHUSD" 
get_candles_all(symbol)

I expect the get_candles_all() method to append "newdata" to "df" iteratively until the final index (timestamp) of df is within 2 mins of targettime.

Continued "ValueError: If using all scalar values, you must pass an index" error despite various attempts to either use non-scalar values, or pass an index.

Answer 1

df.set_index(["MS"], inplace=True) 

or

df = pd.DataFrame(response,index=[value])