Search code examples
c++tokentokenize

Tokenizing string and put tokens in variables of different data types?


i want to tokenize a string e.g. "5 + 3 * 2 - 4 " everytime when a " " (space) appears. the operator signs shall be saved in different named char variables. the numbers in different named float variables. e.g.

float num_1 = 5;
float num_2 = 3;
float num_3 = 2;
float num_4 = 4;

char op_1 = '+';
char op_2 = '*';
char op_3 = '-';

this is my tokenizer so far. but this can only print the numbers and operatos in console. thats not what i want.

std::vector<std::string> strings;
    std::istringstream f("3 + 2 * 4 ");
    std::string s;
    while (std::getline(f, s, ' ')) {
        std::cout << s << std::endl;
        strings.push_back(s);
    }

i googled this stuff for an eternity, but i never found something helpful to me. im just a beginner and most of this code i googled was too hard for me to actually understand or it was just not what i need :/


Solution

  • It seems you have most of the job done. What you're missing is a way to know whether the substring s is an operator or a number. You can easily code helper functions for that. Here's a modified version of your code that should work just well. It's certainly not bulletproof as far as input goes, but it should get you going.

    // Checks if the string is an operator
    bool IsOperator(std::string const& str)
        {
        if (1 != str.size())
            return false;
    
        if (str[0] == '*' || str[0] == '-' || str[0] == '+' || str[0] == '/')
            return true;
    
        return false;
        }
    
    // Converts the string into a float
    float ParseValue(std::string const& str)
        {
        return stof(str);
        }
    
    int main()
        {
        float* = n
        std::vector<float> values;
        std::vector<char> operators;
    
        std::istringstream f("3 + 2 * 4 ");
        std::string s;
        while (std::getline(f, s, ' '))
            {
            if (IsOperator(s))
                operators.push_back(s[0]);
            else
                values.push_back(ParseValue(s));
            }
    
        return 0;
        }
    

    EDIT -- with arrays.

    int main()
        {
        float values[100];
        char operators[100];
    
        size_t valuesIdx = 0;
        size_t operatorsIdx = 0;
    
       std::istringstream f("3 + 2 * 4 ");
        std::string s;
        while (std::getline(f, s, ' '))
            {
            if (IsOperator(s))
                {
                operators[operatorsIdx] = s[0];
                ++operatorsIdx;
                }
            else
                {
                values[valuesIdx] = ParseValue(s);
                ++valuesIdx;
                }
            }
    
        return 0;
        }