MongoDB: IAM user lookups by account and user name scan the iam_users collection #841

Problem

The MongoDB catalog has one index on iam_users: a unique index on user_arn (crates/storage-mongodb/src/migrations/m0001_baseline.rs:99). Three store functions look up a user with the filter { account_id, user_name }:

• get_user_detail (crates/storage-mongodb/src/management_store.rs:483)
• verify_iam_user_password (:630), which every account_id/user_name login to the management API, /metrics and the console calls
• create_access_key (:1724)

No index has these fields, so MongoDB answers each of these lookups with a collection scan. A lookup for a user that does not exist reads every document in iam_users. A lookup for a user that exists stops at the first match.

Two effects follow:

• The time of each of these calls grows with the number of IAM users in the deployment.
• A login for an account_id/user_name that has no user takes longer than a login for one that has, apart from bcrypt. SQLite, PostgreSQL and Cassandra read the user by its primary key.

Not measured: the size of the difference. The measurement is the time of verify_iam_user_password for an existing and a missing user, on a MongoDB 7 or 8 replica set. Repeat it at 1,000, 10,000 and 100,000 IAM users.

Expected

The three lookups use an index. Two ways to get there:

• Filter on user_arn. create_user builds it as arn:aws:iam::{account_id}:user/{user_name} (management_store.rs:337), so the unique index serves the lookup. This needs a check that every stored user has an ARN of that form.
• Add a unique index on { account_id: 1, user_name: 1 } in a catalog migration, as fix(storage-mongodb): store each access key ID once #823 does for access_keys.

A test that runs the lookup with explain and expects an index scan would keep it that way.