MCP Gateway Evaluation Checklist: 25 Checks Before You Commit

Evaluate an MCP gateway by turning claims into tests: issue a scoped key, call a hidden tool by name, trigger an approval-gated action, find the call in the log, revoke the key and read the data terms. This checklist groups 25 checks into seven areas, each with a pass criterion you can verify in an afternoon.

How to use this list

Run every check against every candidate, in a test environment, with the same client and the same sample tools. Record results in one table. If a check cannot be tested, write “not documented” and treat that as a risk, not a pass.

Key takeaway: a gateway’s features page describes intent. A test describes behaviour. Decide on behaviour.

1. Identity and keys

#	Check	Pass criterion
1	Separate credentials per caller	You can create several keys or identities with independent revocation
2	Revocation is immediate	The next call after revocation fails
3	Keys shown once, stored hashed or encrypted	Documentation says so; the console cannot reveal an old key
4	OAuth supported where assistants need it	Documented flow, including PKCE for public clients
5	No token passthrough	A client’s token is never forwarded to an upstream service

For background on these, see MCP API key management. BusinessMCP’s documentation for exposing an MCP endpoint is an example of how a vendor can document the first three points: keys shown once, one per agent, revocable independently.

2. Tool policy

#	Check	Pass criterion
6	Deny by default	A new key sees no tools until granted
7	Discovery filtered per key	tools/list differs by policy
8	Call-time enforcement	A hidden tool called by name is refused
9	Data-class or category policy	Policy can be expressed above individual tool names
10	Catalogue is documented	You can list every tool without being a customer

Point 10 is easy to overlook. A published list lets reviewers assess risk before signing anything. BusinessMCP’s catalogue of MCP gateway tools is an example of one, with categories, plan gates and approval markers. Use the allowlist guide to design your own policy.

3. Approvals and limits

#	Check	Pass criterion
11	Approval for irreversible actions	A gated call waits and does nothing until approved
12	Approval enforced server-side	Cannot be bypassed from the client
13	Per-run or per-key ceilings	Exceeding a limit returns a clear limit response
14	Quotas documented	Plan or usage limits are stated, not implied

4. Logging and audit

#	Check	Pass criterion
15	Per-call record	Caller, tool, arguments, result and time are logged
16	Refusals logged	Denied calls appear
17	Export	Logs can be exported to your store
18	No secrets in logs	Search a test log for your key; it is absent

See MCP audit logging for a field list and an exercise to run.

5. Secrets and isolation

#	Check	Pass criterion
19	Upstream credentials kept server-side	Never appear in tool arguments, output or client config
20	Tenant isolation	Documented design, ideally enforced in the database
21	Outbound request guards	If the gateway fetches URLs, it documents protection against server-side request forgery

For check 21, note how openly a project describes its limits. BusinessMCP’s hardening-utilities repository, for example, includes an SSRF guard but also states that it has had no external audit and that a DNS-rebinding window remains. That honesty is a positive signal.

6. Operations and exit

#	Check	Pass criterion
22	Protocol version support	Documented, with a deprecation policy
23	Monitoring and status	You can see health, errors and latency
24	Data terms	Processing agreement, location, sub-processors and retention are readable
25	Exit plan	You can export policy, keys metadata and logs

7. Threats to test by hand

Beyond the numbered list, spend an hour on adversarial tests. OWASP lists prompt injection and excessive agency among the main risks for LLM applications, and a gateway only partly addresses them.

1. Injected instruction. Put text in a document the agent will read that tells it to call a tool outside its job. Confirm the gateway refuses, or approval stops it.
2. Poisoned description. If you connect a third-party server, review how tool descriptions are shown and whether changes require re-consent.
3. Cross-key access. With a key for one workspace, attempt to reach data from another.
4. Loop. Make an agent call the same tool repeatedly. Do limits trigger?

BusinessMCP’s guide to MCP prompt injection summarises these threats and mitigations if you want a script for the tests.

Running the evaluation in practice

Set aside an afternoon per candidate and prepare in advance. Create a test workspace with synthetic data. Pick four sample tools: one harmless read, one read of something sensitive, one internal write and one that would send something externally, even if the “send” goes to a dummy address. Prepare two clients, one that should be fully permitted and one that should be tightly limited.

Then run the checks in a fixed order so results are comparable. Start with identity because everything depends on it: create the keys, confirm revocation and look at what the console shows after creation. Move to policy and spend most of your time there, since call-time enforcement is the check vendors least often demonstrate. Then approvals and limits, then logs. Leave procurement items such as data terms and exit for the end, because they need people outside the engineering team and take longest.

Keep a simple evidence folder: a screenshot or log excerpt per check, with the date and candidate name. When stakeholders disagree later, evidence ends the argument faster than memory. If a vendor offers to run the demo for you, say yes and then repeat the same steps yourself, since a guided path hides the awkward corners.

Finally, note the experience as well as the results. How long did it take to issue the first scoped key? How clear were the errors? Did the documentation match the product? A gateway that passes every check but takes a day to configure will be configured badly, and a bad configuration is a security failure whatever the feature list says.

Scoring

Give each check 0 (fails or undocumented), 1 (partial) or 2 (passes). Some checks are gates, meaning any zero ends the evaluation: 2, 8, 11 and 19. Totals matter less than the gates. A gateway that scores well overall but fails call-time enforcement should not be used with sensitive tools.

Score range	Reading
40 to 50, no gate failures	Strong candidate; proceed to security review
30 to 39	Usable for low-risk tools; document gaps
Below 30 or any gate failure	Do not use with sensitive data

These ranges are our suggestion, not a standard. Adjust them to your risk appetite.

Mistakes in evaluations

• Testing only the happy path.
• Using production credentials in a trial.
• Letting vendor demos set the agenda.
• Forgetting to test revocation and export until after purchase.
• Ignoring the people cost of self-hosted options. See self-hosted versus managed gateways.

Once you have scores, use the best MCP gateway comparison to shortlist, and revisit this checklist annually.