AI Monitor - Operational Scripts 26h2 2026-07-26
=================================

Prerequisites
-------------
- Windows 10/11 or Windows Server 2016+
- AWS CLI v2.13+ (latest recommended): https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
- AWS credentials configured (aws configure, SSO, or environment variables)
- PowerShell 5.1+ (included with Windows 10/11)
- Python 3.10+


users.cmd / users.sh
--------------------
Manage Cognito users for the AI Monitor editor.

Usage:
  users.cmd <stack-name> <action> [email] [temp-password] [region]

Actions:
  create <email> [temp-password]   Create a new user (prompted to reset on first login)
  list                             List all users
  delete <email>                   Delete a user
  reset <email> [temp-password]    Reset password (temporary, must change on login)
  disable <email>                  Disable a user
  enable <email>                   Re-enable a user

Examples:
  users.cmd AIMonitor create admin@example.com "Temp*88=888Pass123!@#"
  users.cmd AIMonitor list
  users.cmd AIMonitor disable user@example.com


groups.cmd / groups.sh
----------------------
Manage Cognito groups for RBAC (advanced+ tiers).
Users in the "admin" group have full access. Users in the "viewer" group are read-only.
Users not in any group are denied access.

Usage:
  groups.cmd <stack-name> <command> [args] [region]

Commands:
  list-groups                         List all groups
  create-group <name> [description]   Create a new group
  delete-group <name>                 Delete a group
  add-to-group <user> <group>         Add a user to a group
  remove-from-group <user> <group>    Remove a user from a group
  list-users [group]                  List users (optionally in a specific group)

Examples:
  groups.cmd AIMonitor create-group admin "Full access editors"
  groups.cmd AIMonitor create-group viewer "Read-only access"
  groups.cmd AIMonitor add-to-group admin@example.com admin
  groups.cmd AIMonitor list-users admin

sso.cmd / sso.sh
-----------------
Manage external SSO (SAML/OIDC) identity providers on the Cognito User Pool
for federated login to the AI Monitor editor. Enterprise tier only.

Usage:
  sso.cmd <stack-name> <command> [args]

Commands:
  info                                         Show Cognito pool details (ACS URL, Entity ID, Redirect URI)
  setup-saml <name> <metadata-url>             Add a SAML identity provider (Okta, Azure AD, ADFS)
  setup-oidc <name> <issuer> <client-id> <secret>  Add an OIDC identity provider (Google, Auth0)
  list                                         List configured identity providers
  remove <name>                                Remove an identity provider

Workflow:
  1. Run "sso.cmd <stack> info" to get your ACS URL and Entity ID
  2. Configure your IdP (Okta, Azure AD, etc.) with those values
  3. Run "sso.cmd <stack> setup-saml <name> <metadata-url>" to register it
  4. Users see a "Sign in with <name>" button on the Cognito login page
  5. After first login, assign groups: groups.cmd <stack> add-to-group <federated-username> admin

Notes:
  - Federated usernames have format: <ProviderName>_<email> (e.g. Okta_user@domain.com)
  - Group assignments must use the federated username, not the email alone
  - JIT provisioning is automatic (no pre-creation needed in Cognito)
  - Removing a provider immediately prevents SSO login via that provider
  - Native Cognito users are unaffected by IdP changes

Examples:
  sso.cmd AIMonitor info
  sso.cmd AIMonitor setup-saml Okta https://dev-xxx.okta.com/app/yyy/sso/saml/metadata
  sso.cmd AIMonitor setup-oidc Google https://accounts.google.com client-id client-secret
  sso.cmd AIMonitor list
  sso.cmd AIMonitor remove Okta

Requires: AWS CLI, Cognito UserPool deployed, enterprise tier.
See SSOGuide.pdf for detailed IdP setup instructions.
    
setup-domain.cmd / setup-domain.sh
-----------------------------------
Configure a custom domain (e.g. monitor.company.com) for the AI Monitor editor ALB.
Handles ACM certificate creation, DNS validation via Route 53, and ALB alias record.

Usage:
  setup-domain.cmd cert-only <domain> <hosted-zone-id> [region]
  setup-domain.cmd <stack-name> <domain> <hosted-zone-id> [region]

Modes:
  cert-only    Request and validate an ACM certificate without modifying the stack.
               Run this BEFORE deployment if you want HTTPS on a custom domain.
               The certificate must be validated (DNS record in Route 53) before
               it can be used by the ALB.

  <stack-name> Full setup: creates the Route 53 alias record pointing your domain
               to the ALB. Run this AFTER deployment. If no certificate exists,
               it creates one and waits for validation.

Workflow:
  1. (Optional) Pre-create cert: setup-domain.cmd cert-only monitor.company.com Z0123456789
  2. Deploy the stack with the CertificateArn parameter
  3. Create DNS alias: setup-domain.cmd AIMonitor monitor.company.com Z0123456789
  4. Access the editor at https://monitor.company.com/editor

Notes:
  - Certificate must be in us-east-1 (same region as ALB)
  - DNS validation typically takes 1-5 minutes
  - The script waits for certificate validation before proceeding
  - Existing certificates for the same domain are reused

Examples:
  setup-domain.cmd cert-only monitor.perfware.cloud Z04ABCDEF123
  setup-domain.cmd AIMonitor monitor.perfware.cloud Z04ABCDEF123
  setup-domain.cmd AIMonitor monitor.perfware.cloud Z04ABCDEF123 us-east-1

Requires: AWS CLI, Route 53 hosted zone for your domain.
        
support-bundle.cmd / support-bundle.sh
--------------------------------------
Collects diagnostic information for troubleshooting. Outputs a timestamped
bundle directory with Lambda configs, recent logs, DynamoDB state, CloudWatch
metrics, alarm status, discovery cache, AI circuit breaker state, and more.

Usage:
  support-bundle.cmd <stack-name> [region]

Output:
  support-bundle-<timestamp>/ directory with JSON/text files.
  Run analyze-bundle.py on the output for a summary report.

Examples:
  support-bundle.cmd AIMonitor
  python analyze-bundle.py support-bundle-20250115-143022/


setup-cross-account-metrics.cmd / setup-cross-account-metrics.sh
-----------------------------------------------------------------
Run in the REMOTE account to set up cross-account metric delivery into the
central AI Monitor stack's S3 bucket.

Architecture:
  Remote: Metric Stream -> Firehose (+ transform Lambda) -> Central S3
  Central: S3 event -> SQS -> Collector Lambda -> OpenSearch

CloudWatch Metric Streams require Firehose in the SAME account.
Firehose supports cross-account S3 destinations natively.

Prerequisites:
  - Central AI Monitor stack deployed with CrossAccountIds including the remote account,
    OR OrganizationId set to allow any account in the AWS Organization
  - AWS CLI configured for the REMOTE account

Usage:
  setup-cross-account-metrics.cmd <central-account-id> <central-stack-name> <central-region> [options]

  Options:
    -filter=all              Stream ALL metrics (default)
    -filter=discover         Auto-discover active namespaces in this account
    -filter=file://path.txt  Read namespaces from file (one per line)
    --org=<org-id>           AWS Organizations mode (no explicit account IDs needed)
    --status                 Verify existing setup and show resource health
    --remove                 Remove all cross-account resources for this stack

Examples:
  setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1
  setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 -filter=discover
  setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 -filter=file://namespaces.txt
  setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 --org=o-abc123xyz
  setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 -filter=discover --org=o-abc123xyz
  setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 --status
  setup-cross-account-metrics.cmd 000000000000 AIMonitor us-east-1 --remove

The discover option calls 'aws cloudwatch list-metrics' to find all namespaces
with active metrics in the remote account, then configures the Metric Stream
to only include those namespaces. Saves the list to namespaces.txt for reuse.

The --org option is informational — it adjusts the completion message to note
that the central stack uses OrganizationId for S3 bucket policy access rather
than requiring explicit CrossAccountIds. The resources created are the same.

The --status option checks the health of an existing setup:
  - Metric Stream state and last update time
  - Firehose delivery stream status
  - Transform Lambda existence and modification date
  - IAM role existence (firehose, stream, transform, metrics-reader)
  - S3 bucket connectivity test

The --remove option tears down all resources created by this script:
  - CloudWatch Metric Stream
  - Firehose delivery stream
  - Transform Lambda function
  - IAM roles (firehose, stream, transform, metrics-reader)

Example namespaces.txt:
  AWS/Lambda
  AWS/EC2
  AWS/RDS
  AWS/DynamoDB

Creates in the remote account:
  1. Transform Lambda (splits Metric Stream concatenated JSON)
  2. Firehose delivery stream with Lambda processor + cross-account S3 destination
  3. CloudWatch Metric Stream -> local Firehose
  4. Metrics Reader role (allows central account to list metrics in this account)


AWS Organizations Support (advanced/enterprise)
------------------------------------------------
Instead of listing individual account IDs in CrossAccountIds, you can set the
OrganizationId parameter (e.g. o-abc123xyz) on the central stack. This allows
any account in the organization to deliver metrics without updating the stack
each time a new account is added.

Both methods can coexist:
  - OrganizationId: any org member account gets automatic access
  - CrossAccountIds: explicit access for accounts outside the org

When using OrganizationId, remote accounts still need to run the
setup-cross-account-metrics script to create their Firehose and Metric Stream.
The only difference is the central stack's S3 bucket policy uses
aws:PrincipalOrgID instead of explicit account principals.


check-quota.cmd / check-quota.sh
---------------------------------
Validates AWS service quotas and infrastructure readiness before deployment.
Checks VPC endpoints, Lambda concurrency, DynamoDB capacity, S3, SQS, Firehose,
CloudWatch Metric Streams, OpenSearch, and Bedrock availability.

Usage:
  check-quota.cmd [region]

Bedrock checks:
  - Verifies Bedrock Runtime is available in the region
  - Confirms Anthropic Claude model is listed
  - Tests InvokeModel (catches use case form requirement)
  - Reports daily token quota and warns if insufficient

Examples:
  check-quota.cmd
  check-quota.cmd us-east-1


Enabling AWS/Billing Metrics in Remote Accounts
------------------------------------------------
AWS/Billing metrics (EstimatedCharges) are not enabled by default.
Each account that needs billing monitoring must enable this separately.

Steps:
  1. Log in to the AWS Management Console as the root user or an IAM identity
     with billing permissions.
  2. Navigate to the AWS Billing and Cost Management console.
  3. In the left navigation pane, choose Billing preferences.
  4. Find the "Alert preferences" section and choose Edit.
  5. Select the checkbox for "Receive CloudWatch Billing Alerts".
  6. Choose Save preferences (or Update) to apply the changes.

Note: Billing metrics are only published in us-east-1 regardless of which
region your resources are in. The Metric Stream in the remote account must
be in us-east-1, or you must create a separate stream in us-east-1 for
AWS/Billing if your primary stream is in another region.

After enabling, AWS/Billing EstimatedCharges metrics will begin publishing
within a few hours. These can then be streamed cross-account like any other
CloudWatch metric.
