AI SIEM Helper Scripts 26h4b 2026-10-22
============================================

Windows .cmd and Linux/macOS .sh versions are provided for most tasks; a few run
directly in Python and need boto3 (pip install boto3). Run each with AWS
credentials for the account named in its row of the SCRIPT REFERENCE below.
Every script prints its own usage when run with no arguments.

Commands below are written scripts\name.cmd for clarity. On Linux/macOS use the
matching scripts/name.sh (same arguments). If you run them from inside the
unzipped bundle folder, drop the scripts\ prefix.


CONNECTING DATA SOURCES (START HERE)
------------------------------------
AI SIEM takes data three different ways. Knowing which applies tells you whether
there is anything to wire.

  1. S3 LOG BUCKETS  -  CloudTrail, VPC Flow Logs
     Delivered as files to an S3 bucket. AI SIEM does not poll; the bucket must
     send s3:ObjectCreated notifications to the ingestion queue.
       - CloudTrail: enter the bucket in the editor (Configure -> Data Sources).
         The editor wires the notification for you when it can. If it reports it
         could not (usually a cross-account bucket), use wire-log-bucket (below).
       - VPC Flow Logs: the stack already owns the flow-log bucket (policy,
         notification, Glue table). Run create-flowlogs to turn on flow logs for
         a WORKLOAD VPC you name, delivered to that stack bucket.

  2. AWS-NATIVE EVENTS  -  GuardDuty, Security Hub, AWS Config
     Arrive over EventBridge via rules the stack already created. In the SIEM's
     OWN account nothing is wired. They only produce data when the underlying
     AWS service is enabled and something happens, so zero events on a quiet
     account is normal.
       - CROSS-ACCOUNT: EventBridge findings do NOT cross accounts on their own.
         To bring another account's GuardDuty/Security Hub/Config findings in,
         run wire-events in that account (see CROSS-ACCOUNT EVENTBRIDGE FINDINGS
         below). The central stack must also have been deployed with
         OrganizationId or CrossAccountIds so its event bus accepts them.

  3. PLATFORM CORRELATION  -  AI Monitor, Log Processor
     AI Monitor writes metric anomalies into the shared OpenSearch domain; AI
     SIEM reads them and correlates. Log Processor pattern detections reach the
     SIEM the same way, through AI Monitor. There is NO bucket and NO script for
     this: in the editor (Configure -> Data Sources -> AI Monitor Correlation)
     pick which subscriptions contribute. That is the whole setup.

Case (1) always needs a helper for cross-account, and case (2) needs one only
for cross-account EventBridge findings. Same-account (2) and all of (3) are
wiring-free.


CROSS-ACCOUNT LOG BUCKETS (READ THIS BEFORE WIRING ONE)
-------------------------------------------------------
A log bucket in ANOTHER account (a common setup: an organization CloudTrail in a
log-archive account) needs FOUR things, and getting only some of them produces a
silent failure that is easy to misread as "it works":

  1. Notification: the bucket sends s3:ObjectCreated to the ingestion queue.
  2. Queue policy: the ingestion queue allows the bucket's account to send.
       -> wire-log-bucket does BOTH of these. Run it with the profile for the
          account that OWNS the bucket (4th positional arg):
            wire-log-bucket.cmd AISIEM org-trail-logs us-east-1 logarchive

  3. Bucket policy: must allow <stack>-IngestionRole to s3:GetObject on the
     objects. The role already permits it on its side, but a cross-account read
     also needs the bucket policy to allow it.
  4. KMS key policy (only if the bucket is SSE-KMS encrypted): must allow that
     same IngestionRole to kms:Decrypt.

wire-log-bucket does 1 and 2. It does NOT do 3 and 4 - those must be applied by
the bucket's owner. WITHOUT 3 and 4, notifications still arrive and then every
object read fails: the queue drains into the DLQ and nothing is indexed, even
though the source shows as enabled. If a cross-account source looks connected
but no events appear, this is almost always why. As an alternative to 3 and 4,
replicate the bucket into one the SIEM account owns and wire that.

Same-account buckets need only step 1 (the editor or wire-log-bucket handles it);
the queue policy already covers same-account, and no bucket/KMS grant is needed.

CloudTrail on a SHARED bucket: an organization CloudTrail already delivers every
member account to one bucket under AWSLogs/<account-id>/CloudTrail/..., which
ingestion tags per record - so one wire-log-bucket call (prefix AWSLogs/) covers
the whole org. If the SAME bucket also carries flow logs, wire each source on its
own non-overlapping prefix (see PREFIXES ON A SHARED BUCKET under VPC FLOW LOGS).
The four cross-account grants above apply to the CloudTrail bucket exactly as to
any other cross-account log bucket.


CROSS-ACCOUNT EVENTBRIDGE FINDINGS (GuardDuty, Security Hub, Config)
--------------------------------------------------------------------
GuardDuty, Security Hub and Config deliver findings to EventBridge only in the
account and region where the service runs. The stack's own rules therefore see
findings from the SIEM account only. To bring in another account's findings,
forward them to the SIEM's event bus. Two sides:

  CENTRAL (SIEM) side - one-time, at deploy:
    Deploy the stack with OrganizationId (any account in the org may forward) OR
    CrossAccountIds (only the listed accounts may forward). This emits the bus
    policy that permits events:PutEvents onto the SIEM's default event bus. The
    policy is source-agnostic, so it covers all three sources at once. Setting
    NEITHER leaves the bus closed - single-account deployments need nothing.

  SOURCE side - run wire-events in the source account:
      wire-events.cmd setup <central-account-id> [region] [--source LIST]
    --source is a comma list of guardduty,securityhub,config (default: all). It
    creates one forwarder role EventBridge assumes and one rule per source,
    sending findings to the central account's default bus. Forwarded events keep
    their source and originating accountId, which ingestion partitions on.

    wire-events.cmd remove [region] [--source LIST]   tears rules down (and the
                                                      shared role when the last
                                                      rule is removed).
    wire-events.cmd status [region] [--source LIST]   shows current wiring.

WHERE TO RUN IT (per source - you may not need it in every account):
  GuardDuty    Run in each source account+region.
               EXCEPTION - delegated administrator: if a source account is your
               org's GuardDuty delegated admin AND the SIEM runs in it, member
               findings already aggregate there and the stack's own rule sees
               them. wire-events detects central == local and no-ops.
  Security Hub If you use Security Hub finding AGGREGATION (an aggregation region
               in an org), all member findings are re-imported into the
               aggregator account already. Run wire-events ONCE in the aggregator
               account - not in every member. If the SIEM IS the aggregator, do
               nothing. Without aggregation, run per source account.
  Config       Run in each source account+region. Forwards Config RULE
               compliance changes only, so the account must actually have Config
               rules that evaluate (Config recording with no rules emits nothing
               to forward). Full Config configuration SNAPSHOTS are a separate,
               S3-delivery dataset and are not ingested today.


MULTI-ACCOUNT SETUP
-------------------
The multi-account model uses cross-account IAM roles for read access to
CloudTrail and VPC Flow Log S3 buckets in remote accounts, plus an optional
response role for enterprise-tier remediation. Advanced or Enterprise tier
required.

Flow:
  1. Remote account: setup-remote-access.cmd setup <central-acct-id> [region] [--org <org-id>]
     Creates AISIEM-remote-read and AISIEM-remote-respond roles. Pass --org
     <org-id> to also trust any account in that AWS Organization.
  2. Central account: add-account.cmd <stack> <remote-acct-id> --trail-bucket <bucket>
     Registers the account so the ingestion Lambda starts reading from it.
  3. Ingestion Lambda assumes AISIEM-remote-read to list/read S3 objects.
  4. Response Lambda assumes AISIEM-remote-respond for remediation (enterprise).

You can also list account IDs in the CrossAccountIds stack parameter, or set
OrganizationId to authorize an entire AWS Organization. Either may be used, or both.

MULTI-REGION COVERAGE
  CloudTrail    An organization trail delivers all regions to one bucket, so a
                single bucket covers everything.
  VPC Flow Logs Regional. Configure flow logs in each region to deliver to one
                central bucket.
  GuardDuty /   EventBridge is regional and per-account. Use a delegated
  Security Hub  administrator with auto-enable across regions (findings aggregate
                to the admin account), or forward with wire-events per source
                account+region.


VPC FLOW LOGS
-------------
VPC Flow Logs are the signal for network-based detections: port scanning,
blocked connections, and (with ALL traffic) large-transfer exfiltration. They
are not on by default because they bill per GB delivered and stored.

The stack ALREADY owns the flow-log bucket. When you deploy AI SIEM it creates a
hardened bucket (lifecycle, encryption, the delivery policy for local AND
cross-account VPCs, the notification to the ingestion queue, and the Athena Glue
table flowlogs_raw over it). create-flowlogs does NOT create or wire a bucket; it
just turns on a flow log for a VPC you choose, pointed at that stack bucket, with
per-hour Hive partitioning so Athena can query it directly:

  set AWS_PROFILE=<your-profile>
  scripts\create-flowlogs.cmd <stack> --vpc <vpc-id>
  scripts\create-flowlogs.cmd <stack> --vpc-from-stack <other-stack> --traffic REJECT

Re-running is harmless (every step is idempotent). Flow-log delivery is
aggregated, so the first objects can take up to ~10 minutes to appear. The
editor's Data Sources panel shows the last event time once ingestion begins.

  --vpc / --vpc-from-stack   REQUIRED. Name the workload VPC to monitor, either
                             by id or by another stack's VpcId output. There is
                             no default: this tool never enables flow logs on
                             the shared LogProcessor / AI Monitor VPC that the
                             SIEM itself runs in (that would create a
                             self-surveillance loop), and it refuses to run if
                             the target resolves to that VPC.
  --traffic ALL|REJECT|ACCEPT   ALL (default). S3 delivery is cheap even at ALL
                             volume, and capturing everything is what lets the
                             on-demand Athena flow-query pull exfiltration detail
                             around an incident. Use REJECT only to minimize even
                             S3 storage (you then lose ALL-traffic forensics;
                             port-scan detection still works).
  --bucket <name>            Override the destination bucket. Defaults to the
                             stack's FlowLogsBucket output; the main use is
                             cross-account (run from a remote account and pass
                             the central stack's bucket).
  --region <region>          Default us-east-1.
  --profile <name>           AWS profile.
  --dry-run                  Preview every change without writing.

ONE OR MANY VPCs, ONE BUCKET
  Point as many VPCs as you like (any account, any region) at the one stack
  bucket. AWS partitions deliveries under a Hive key path
  (AWSLogs/aws-account-id=<id>/aws-service=vpcflowlogs/aws-region=<region>/
  year=/month=/day=/hour=/...), so they never collide and the flowlogs_raw Glue
  table's partition projection resolves them all. Run create-flowlogs once per
  VPC; every step is idempotent.

CROSS-ACCOUNT FLOW-LOG DELIVERY (honors CrossAccountIds / OrganizationId)
  The stack bucket's policy already authorizes cross-account delivery, set at
  deploy time from the central stack's own parameters -- there is no bucket
  policy for this helper to edit:
    - OrganizationId set  -> the whole org may deliver (aws:SourceOrgID). No
                             account list to maintain.
    - CrossAccountIds set -> each listed account may deliver.
  To onboard a remote VPC: run create-flowlogs FROM that account (with its own
  credentials, since a flow log is created where the VPC lives) and pass
  --bucket <central stack's FlowLogsBucket>. If the account is not covered by
  OrganizationId / CrossAccountIds, redeploy the central stack with it added first.

PREFIXES ON A SHARED BUCKET
  wire-log-bucket keys each notification by prefix, so one bucket can carry
  several sources (e.g. cloudtrail/ and vpcflowlogs/) with distinct, coexisting
  notifications - wiring a second prefix does not overwrite the first. Wire each
  with its own --prefix. One caveat from S3: notification prefix filters for the
  same event type may not OVERLAP. Do not wire both a parent prefix (AWSLogs/)
  and a child (AWSLogs/111/) on one bucket; use non-overlapping prefixes.


AUDIT TRAIL (WORM AUDIT BUCKET)
-------------------------------
Every threat status change (Acknowledge, Investigate, Resolve, False Positive,
Dismiss) and every remediation action is recorded with WHO did it and WHEN, in
two places:
  - DynamoDB (the fast path the editor reads to show "Audit History" per threat).
  - A dedicated S3 bucket <stack>-siem-audit-<acct>-<region> with S3 Object Lock
    (WORM), as the tamper-evident system of record. Each record is a write-once
    object: the application roles can APPEND audit records but have no permission
    to alter or delete them, so even a compromised Lambda cannot rewrite history.

The lock behavior is chosen at deploy time with the AuditLockMode parameter:

  GOVERNANCE (default)
    Write-once, but a principal explicitly holding s3:BypassGovernanceRetention
    (typically an account admin) can still delete a record early. This leaves a
    controlled decommission path. Recommended for most deployments.

  COMPLIANCE
    Fully immutable. NO ONE - not the application, not an admin, not the account
    root - can alter or delete a record before its retention date elapses. This
    is what strict regimes (e.g. SEC 17a-4-style requirements) need.

    !! IMPORTANT CONSEQUENCE OF COMPLIANCE MODE:
       An S3 bucket can only be deleted when it is empty, and COMPLIANCE-locked
       objects CANNOT be emptied until their retention passes. So under
       COMPLIANCE the audit bucket - and the S3 storage cost for it - cannot be
       removed until the LAST audit record's retention has elapsed. Because this
       makes AuditRetentionDays an IRREVERSIBLE commitment, set it to your actual
       regulatory requirement - not higher "to be safe". Under GOVERNANCE an
       over-long retention is recoverable via bypass; under COMPLIANCE it is not.

Retention length: AuditRetentionDays (default 365; e.g. 2555 for 7 years).
Applies to newly written records; existing records keep the mode and retention
they were written under (you cannot retroactively weaken a lock).

The audit bucket has DeletionPolicy: Retain - a CloudFormation stack delete
leaves it in place (it is the system of record and must outlive the stack).


TEARDOWN (REMOVING AI SIEM)
---------------------------
Deleting the stack does NOT stop your log sources. Bucket notifications and any
VPC flow logs created for AI SIEM live outside CloudFormation and survive the
delete. Left in place, buckets keep firing notifications at a queue that no
longer exists, and flow logs keep writing to S3 while you keep paying for it.

Run in this order:

  1. python scripts\sever_sources.py <stack> [region]
       Dry run, the default. Changes nothing. Reports:
         - VPC flow logs delivering to S3
         - bucket notifications owned by this stack
         - the CloudTrail trail and whether it is still logging
         - Lambda reserved concurrency (reported, never changed)
       Read this output before going further.

  2. python scripts\sever_sources.py <stack> [region] --apply
       Confirms each item individually. Deletes the flow logs, removes this
       stack's bucket notifications (merge-safe: only Ids belonging to this stack
       are dropped; anything another tool put there is written back untouched),
       and STOPS the trail.

       Flags:
         --delete-trail            Delete the trail instead of stopping it. A
                                   multi-region trail is an audit control and
                                   deleting it breaks CloudTrail history
                                   continuity for the whole account. Stopping is
                                   the default for that reason.
         --bucket-profile <name>   For buckets in another account. Use the same
                                   profile you passed to wire_log_bucket.py, once
                                   per account.
         --delete-buckets          IRREVERSIBLE. Also empty and delete the log
                                   buckets AI SIEM created. Deliberately narrow: a
                                   bucket qualifies only if its name starts with
                                   aisiem-cloudtrail-, or it is the destination of
                                   a flow log or trail found in step 1. The
                                   stack-owned flow-log bucket is never a candidate
                                   (CloudFormation deletes it with the stack). A
                                   bucket that merely carries one of our
                                   notifications - most likely YOUR archive - is
                                   never deleted. Each bucket prints its version
                                   and delete-marker count and size, then requires
                                   you to type the name back before anything is
                                   destroyed.

  3. Delete the CloudFormation stack
       aws cloudformation delete-stack --stack-name <stack> --region <region>
       aws cloudformation wait stack-delete-complete --stack-name <stack> --region <region>
       Buckets with content and any resource still in use can hold the delete up;
       the stack events tab names whatever is blocking.

  4. OpenSearch artifacts - no action needed. The stack's custom resources remove
     AI SIEM's dashboards, index templates, ISM policy, and seed indices from the
     shared domain automatically on delete. (Your event data lives in the S3
     datalake, not the domain.)

THINGS NO SCRIPT DOES - DECIDE DELIBERATELY
  - Log buckets you wired in yourself are never deleted; verify with `aws s3 ls`
    that what you expect to keep is still there.
  - Cross-account roles created by setup-remote-access (AISIEM-remote-read,
    AISIEM-remote-respond) remain in each remote account. Remove them there with
    `setup-remote-access.cmd remove <central-account-id>`.
  - Source-account forwarding set up with wire-events (the AISIEM-*-forward rules
    + AISIEM-eventbridge-forward-role) lives in those remote accounts and
    survives - remove it with `wire-events.cmd remove [region]` in each.
  - The WORM audit bucket (<stack>-siem-audit-*) ALWAYS survives the stack delete
    (DeletionPolicy: Retain). Decommissioning it is a separate, deliberate step -
    under COMPLIANCE it cannot be removed until every record's retention has
    elapsed. See the AUDIT TRAIL section.
  - The delegated remediation add-on, if deployed, created a separate
    aisiem-remediation-<stack> stack in each account where remediations execute.
    Tear it down per account with
    scripts\remediation_docs\deploy-remediation.cmd <stack> --cleanup - see
    remediation_docs\README.md.


QUERYING HISTORICAL DATA
------------------------
Every event is written to an S3 datalake in Parquet, partitioned by account_id
and region for efficient Athena queries and per-account lifecycle, regardless of
whether it was promoted to OpenSearch:

  s3://<stack>-siem-datalake-<acct>-<region>/
    cloudtrail/account_id=.../region=.../year=.../month=.../day=.../*.parquet
    flowlogs/account_id=.../region=.../year=.../month=.../day=.../*.parquet
    guardduty/ securityhub/ config-changes/ threats/

A Glue database named <stack>_siem is created with tables over these prefixes.
Filter on the partition columns so Athena scans only what it needs:

  SELECT * FROM cloudtrail
  WHERE account_id = '111111111111'
    AND region = 'us-east-1'
    AND year = '2026' AND month = '08';


SCRIPT REFERENCE
----------------
  check-quota.cmd / check_quota.py   AI SIEM acct. Verify service quotas before
                                     deploying (Lambda concurrency, per-VPC
                                     interface-endpoint headroom).
  setup-domain.cmd/.sh               AI SIEM acct. Editor custom domain. cert-only
                                     issues + validates the ACM certificate before
                                     launch; <stack> mode creates the Route 53
                                     alias to the ALB after deploy.
  create-flowlogs.cmd/.sh,           AI SIEM acct. Enable VPC Flow Logs on a
    create_flowlogs.py               WORKLOAD VPC, delivered to the stack-owned
                                     flow bucket. Requires a target VPC; never
                                     touches the shared VPC.
  wire-log-bucket.cmd/.sh,           Either. Attach a log bucket (CloudTrail or
    wire_log_bucket.py               Flow Logs) to the ingestion queue when the
                                     editor cannot - notification + queue policy
                                     (merge-safe). Supports --prefix for shared
                                     buckets.
  setup-remote-access.cmd/.sh        Remote acct. Create the cross-account trust
                                     roles (setup/remove/status; --org).
  add-account.cmd/.sh                AI SIEM acct. Register a remote account and
                                     its buckets (<id> / list / remove).
  wire-events.cmd/.sh                Source acct. Forward that account's GuardDuty
                                     / Security Hub / Config findings to the
                                     central event bus (setup/remove/status,
                                     --source LIST).
  users.cmd/.sh                      AI SIEM acct. Cognito users: create, list,
                                     delete, reset password, enable/disable.
  groups.cmd/.sh                     AI SIEM acct. Cognito groups for role-based
                                     access: init, add/remove, list (enterprise).
  sso.cmd/.sh                        AI SIEM acct. Configure SAML or OIDC identity
                                     providers (enterprise).
  support-bundle.cmd/.sh             AI SIEM acct. Collect counts-only diagnostics
                                     for support (no log content or identifiers).
  sever_sources.py                   AI SIEM acct. Detach log sources - the inverse
                                     of wire_log_bucket. First teardown step.
  remediation_docs/                  SELF-CONTAINED SUBFOLDER for delegated auto-
                                     remediation. It has its OWN files (the add-on
                                     template, the sample runbook YAMLs, the deploy
                                     helpers) and its OWN README.md with the full
                                     walkthrough - read that README before running
                                     these. Run them in the account where
                                     remediations EXECUTE:
                                       deploy-remediation.cmd/.sh <siem-stack>
                                         Creates the AutomationAssumeRole SSM
                                         Automation assumes to run your runbooks
                                         (you scope its permissions) + publishes its
                                         ARN to SSM so the editor auto-fills it.
                                         Teardown: same cmd/sh with --cleanup.
                                       deploy-sample-runbooks.cmd/.sh [region] [--privileged]
                                         Creates the no-op test runbooks
                                         (AISIEM-Ping, AISIEM-TestRemediation);
                                         --privileged also adds the isolate /
                                         attach-deny samples.
                                     AI SIEM only calls ssm:StartAutomationExecution
                                     + reads status - it holds no IAM/EC2 power; SSM
                                     runs everything under YOUR role.


SUPPORT
-------
Upload support bundles using the link provided on the landing and fulfillment
pages. Documentation: https://perfware.cloud
