For starters, you should make sure that the agency has worked with enterprise-level clients before. RankScience, for example, has worked with companies like CareerKarma and Coderwall that have massive sites.
Moreover, you’ll want to make sure that your agency can provide custom strategies for your team. Some may try to “rinse and repeat” strategies across various clients—which simply doesn’t work. If your SEO partner doesn’t take the time to understand your SEO goals and audit your existing progress, then that’s a red flag.
Other criteria to consider: innovation, ability to keep up with trends, pricing, and communication.