A Brutal Look at Balanced Parentheses, Computing Machines, and Pushdown Automata

https://en.wikipedia.org/wiki/Dyck_language

  bool balanced(const string& text, const string& open, const string& close) {
    size_t length = text.size(), brackets = open.size();
    assert(close.size() == brackets);
    stack<char> buffer;
    for (size_t index = 0; index < length; ++index) {
      char ch = text[index];
      for (size_t slot = 0; slot < brackets; ++slot) {
        if (ch == open[slot])
          buffer.push(ch);
        else if (ch == close[slot]) {
          if (buffer.empty() || buffer.top() != open[slot])
            return false;
          buffer.pop();
        }
      }
    }
    return buffer.empty();
  }

So to generalise your point you need a counter for each transition to a different type of opener.

So (([])) needs only 2 counters, not 3.

You could constrain it further if certain types of openers are only valid in certain cases so you could exclude certain types of transitions.

EDIT:

([)] could indeed be handled by just additionally tracking the current open type. (([]]) is a better example, as it shows that to handle deeper nesting you need additional pieces of data that will grow at some rate (at most by the number of opens, possibly lower depending on which types can validly appear within which types)

EDIT: It gets complicated if you need to count multiple different types of openers. In that case I think you need the stack, at least unless there are constraints on which openers can occur within others - you at the very least need to know which closer you're looking for right now, but if you can't deduce what is outside, you obviously then need to keep track of it.

In practice, of course, we'll generally use a stack because it's just pointless to make life harder by not using one for this.