#Requires -Version 7.0 # PIM Manager 3.0.70 -- Initialize-PimSqlAdminGroup.ps1, standalone (built by tools/setup/Build-PimSupportScripts.ps1; helpers inlined: ..\..\engine\_shared\pim-rest.ps1, pim-loopbackanswer.ps1, _pimsignedin.ps1, _pimsqladmingroup.ps1, _pimsigninidentity.ps1, ..\..\engine\_shared\pim-installnames.ps1, _pimscriptdoc.ps1) <# .SYNOPSIS Make a role-assignable security group (default grp-pim-sql-admins) the Microsoft Entra admin of a PIM Manager environment's Azure SQL server, with the environment's managed identities and the nightly updater as its members. The Invardia Support app is never added. One command, idempotent, every change read back; -WhatIf shows every change first. Signs in in the browser as the person running it (or, inside an Invardia support session, as the Invardia Support app), or uses a certificate identity from the repository -- never a client secret, no other tools or modules. .DESCRIPTION Makes a security group (default 'grp-pim-sql-admins') the SQL server's Microsoft Entra admin, with these members: * the environment's user-assigned identity (id-pim-; found in the resource group, or named) * the SQL identity id-pim-sql-, when the environment has one (private SQL) * the nightly updater's system-assigned identity (ca-pim-update), when the job exists * the application a build runs as (-SetupIdentityAppId), for the length of that build * any -ExtraMemberObjectIds * the CURRENT admin, so converging never takes access away from whoever has it now The Invardia Support app is NOT a member: its one standing database right is its read-only database user (Grant-PimSupportAccess.ps1); admin rights only inside an open support session. Named with -SupportAppId or -SupportObjectId, it is not added and the run says so. A membership an earlier version gave is left as found. Entra-only authentication is left as it is and verified afterwards. Existing contained database users are not touched. Every change is read back; a second run reports no changes. -ReleaseOwnMembership takes the APPLICATION this run signs in as (-ClientId with -CertThumbprint, or the Invardia Support session) out of the group again, and reads it back. Only that one membership: no other member is touched. The one-shot MSP build uses it at its end when its setting releaseOwnBuildMembership is on (off by default). Why: the unattended updater checks the database schema before it rolls a release. It can only do that if its managed identity can log in, and a group that holds every identity that must administer the store is the one arrangement that survives an identity being added later without a database login being created by hand. See docs/DESIGN.md (the SQL admin group) and the framework contract in the repository's DOCS/REQUIREMENTS.md section 4.5. .PARAMETER SupportAppId The application (client) id of the Invardia Support app. It is NOT added as a member (no standing database administrator); the run says so. The old name -TroubleshootingAppId still works. .PARAMETER SupportObjectId The Invardia Support app's service principal object id, instead of -SupportAppId (also not added). The old name -TroubleshootingObjectId still works. .PARAMETER SetupIdentityAppId The application (client) id a build runs as: added as a member for the length of that build. .PARAMETER ReleaseOwnMembership Take the application this run signs in as out of the group and read it back; nothing else is changed. Needs -ClientId with -CertThumbprint (or the Invardia Support session in this window). .PARAMETER MembersOnly Do not create the group or move the admin: only add the members, and only when the group already IS the server's admin. This is what Deploy-PimUpdateJob.ps1 does on every run. .PARAMETER PlanOnly Read everything and print the plan. Changes nothing. (-WhatIf does the same and prints each change as a "What if:" line.) .PARAMETER TenantId The tenant of the environment. Required for the browser sign-in (the sign-in and every token are pinned to it). .PARAMETER ClientId Automation from the repository only: with -CertThumbprint, a certificate identity in the local store instead of the browser sign-in. Not needed for the downloaded script. .EXAMPLE # A browser window opens: sign in as yourself (an administrator of the tenant). .\Initialize-PimSqlAdminGroup.ps1 -SubscriptionId -ResourceGroup -SqlServerName ` -TenantId -SupportAppId -WhatIf Preview, then run the same command without -WhatIf. .NOTES Requirements: PowerShell: 7.0 or later (pwsh). Windows PowerShell 5.1 is not supported. Role: Privileged Role Administrator (Microsoft Entra; the group is role-assignable); Contributor or Owner on the SQL server and Reader on the environment's resource group (Azure). Hosts: login.microsoftonline.com, graph.microsoft.com, management.azure.com PERMISSIONS AND CHANGES (from Initialize-PimSqlAdminGroup.ps1 manifest, PIM Manager 3.0.70; full page: https://invardia.com/docs/pim/scripts/Initialize-PimSqlAdminGroup/) The person running it needs: - Privileged Role Administrator [entraRole, tenant] -- The group is role-assignable (tier 0): only a Privileged Role Administrator or a Global Administrator can create such a group or change its members. Without it the group is not created and the current SQL admin is left as it is. - Contributor [azureRbac, the Azure SQL server (or its resource group); Owner works too] -- Sets the server's Microsoft Entra admin (Microsoft.Sql/servers/administrators/write) and reads it back. - Reader [azureRbac, the environment's resource group] -- Finds the environment's user-assigned identities (id-pim-*) and the updater job's identity. - The person running it, signed in in the browser (or the Invardia Support app inside an Invardia support session) [other, the subscription named with -SubscriptionId] -- The script takes its Microsoft Graph and Azure Resource Manager tokens from that sign-in (pinned to -TenantId) and refuses a token for another tenant. A certificate identity (-ClientId + -CertThumbprint) is for automation from the repository only; it then needs the Graph application permissions Group.Create and GroupMember.ReadWrite.All (or Group.ReadWrite.All), Directory.Read.All and RoleManagement.ReadWrite.Directory. What it creates, changes or grants: - create security group grp-pim-sql-admins (or -GroupName), role-assignable (isAssignableToRole = true), only when it does not exist (Microsoft Graph) @ tenant -- One group as the SQL admin means every identity that must administer the store gets access by membership, without a database login created by hand. Role-assignable so that a Groups Administrator, a User Administrator or a group owner cannot add themselves to it. Undo: Delete the group in the Entra admin center (Groups) -- only after another Microsoft Entra admin has been set on the SQL server, or nobody can administer the database. - grant member of grp-pim-sql-admins: the environment identity (id-pim-), the SQL identity (id-pim-sql-, private SQL only), the nightly updater's identity (ca-pim-update), a build's own application (-SetupIdentityAppId) for the length of that build, every -ExtraMemberObjectIds, and the previous SQL admin (kept, so nobody loses access). Never the Invardia Support app: named with -SupportAppId / -SupportObjectId it is not added, and the run says so. (Microsoft Graph): group member of the SQL server's Microsoft Entra admin group = administrator of the SQL server and every database on it @ the environment's Azure SQL server -- The updater checks the database schema before it rolls a release, and a setup or build administers the store while it runs. Undo: Remove the member from the group (Entra admin center > Groups > grp-pim-sql-admins > Members). Removing the PIM identities stops PIM Manager. - remove with -ReleaseOwnMembership: the application this run signs in as (-ClientId with -CertThumbprint, or the Invardia Support session) leaves grp-pim-sql-admins; read back. No other member is touched, and nothing else is changed. (Microsoft Graph) @ the environment's Azure SQL server -- A build's identity is a member for the length of the build only. Undo: Run the script again with -SetupIdentityAppId . - update Microsoft Entra admin of the Azure SQL server (Microsoft.Sql/servers/administrators/ActiveDirectory) set to the group (Azure Resource Manager) @ the Azure SQL server named with -SqlServerName -- Azure SQL allows exactly one Microsoft Entra admin; making it the group gives every member access. Entra-only authentication is not changed (only read before and after). Existing contained database users are not touched. Undo: Set the previous admin back (the script prints it): Azure portal > the SQL server > Microsoft Entra ID > Set admin. -WhatIf: Signs in and reads everything, prints the plan, then one "What if:" line per change it would make: create the role-assignable group, add each member (by name and object id), make the group the SQL server's Microsoft Entra admin (and who the admin was). Nothing is created or changed. Undo: Set the previous Microsoft Entra admin back on the SQL server, then remove the members the script added (or delete the group if the script created it). Nothing else is changed by the script. .LINK https://invardia.com/docs/pim/scripts/Initialize-PimSqlAdminGroup/ #> [CmdletBinding(SupportsShouldProcess)] param( [Parameter(Mandatory)][string]$SubscriptionId, [Parameter(Mandatory)][string]$ResourceGroup, [Parameter(Mandatory)][string]$SqlServerName, [string]$SqlResourceGroup, [string]$TenantId, [string]$GroupName = 'grp-pim-sql-admins', [string]$GroupObjectId, [string]$EnvironmentIdentityName, [string]$SqlAdminIdentityName, [string]$UpdateJobName = 'ca-pim-update', # The Invardia Support app. The old names (-TroubleshootingAppId / -TroubleshootingObjectId) still bind. [Alias('TroubleshootingAppId')][string]$SupportAppId, [Alias('TroubleshootingObjectId')][string]$SupportObjectId, # the application a build runs as: a member for the length of that build [string]$SetupIdentityAppId, [string[]]$ExtraMemberObjectIds = @(), # Certificate auth from the local store. Omit both to sign in in the browser (or use the Invardia support session). [string]$ClientId, [string]$CertThumbprint, [switch]$MembersOnly, [switch]$DoNotKeepCurrentAdmin, [switch]$PlanOnly, # take the application this run signs in as out of the group again (its OWN membership only), read back [switch]$ReleaseOwnMembership ) $ErrorActionPreference = 'Stop' # ===== BEGIN inlined helper: ..\..\engine\_shared\PIM-Rest.ps1 ===== <# PIM4EntraPS -- pure-REST auth + data plane (NO Graph/Az/MSAL modules). One place the whole solution gets tokens and calls Microsoft REST APIs, so the engine runs identically on Windows PowerShell 5.1, PowerShell 7, a VM, or a Linux container -- with nothing to Install-Module and no Azure.Core/Graph version clash. Token acquisition (auto-detected, override per call): * Managed Identity -- App Service / Container Apps ($env:IDENTITY_ENDPOINT + IDENTITY_HEADER), an Azure Arc-enabled server (IDENTITY_ENDPOINT + IMDS_ENDPOINT, 401 key-file challenge) or IMDS (VM) * Client secret -- SPN client_credentials (v2 token endpoint) * Client certificate-- SPN, signed RS256 JWT client_assertion (no secret, no MSAL) * Signed-in source -- a registered $global:PIM_TokenProvider (tools\setup\_PimSignedIn.ps1): the Invardia Support session, else the person's browser sign-in (auth code + PKCE). No CLI, no module. Resolution order for credentials (all overridable via params): explicit params -> $global:PIM_* -> Managed Identity -> the signed-in token source -> (opt-in) browser sign-in. Data plane: Invoke-PimGraph / -PimArm / -PimPowerBI / -PimRest with @odata/nextLink paging (-All) and 429/Retry-After backoff. PS 5.1 + 7 compatible (cert signing uses X509 GetRSAPrivateKey, NOT RSA.ImportFromPem which is PS7-only). #> Set-StrictMode -Off # resource (audience) per logical API. Built by a FUNCTION so any scope can re-seed it -- # see the trap note in Resolve-PimRestResource: a dot-sourced function's $script: follows # the CALLER's scope, so a script that loaded PIM-SqlStore without this file found these # tables null and silently ended up with no SQL credential at all. function Get-PimRestResourceMap { @{ graph = 'https://graph.microsoft.com' arm = 'https://management.azure.com' powerbi = 'https://analysis.windows.net/powerbi/api' fabric = 'https://api.fabric.microsoft.com' # PIM 100.142b: Fabric workspace role assignments (the engine adds the PIM group) defender = 'https://api.securitycenter.microsoft.com' exo = 'https://outlook.office365.com' # Exchange Online REST admin API (app-only ManageAsApp) } } $script:PimRestResources = Get-PimRestResourceMap $script:PimTokenCache = @{} # "audience|tenant|clientId|credentialKind" -> @{ token; expiresUtc } function Clear-PimRestTokenCache { <# Drop every cached token. The cache is keyed by full identity (BUG-22), so this is NOT needed to switch tenants any more -- it exists for an explicit "forget everything" (a rotated credential mid-process, or a test). #> [CmdletBinding()] param() $script:PimTokenCache = @{} } function Add-PimRestSessionToken { <# 97.1 -- hand a token the CALLER already holds (a setup script's browser sign-in: auth code + PKCE, no modules) to the token cache, so every Invoke-PimGraph / Invoke-PimArm / Invoke-PimRest / SQL call in THIS process uses it. Only for the attended interactive mode ($global:PIM_Interactive): the key is exactly the one Get-PimRestToken computes for that mode, so the engine's own identities (managed identity, certificate) can never pick it up. #> [CmdletBinding()] param([Parameter(Mandatory)][string]$Resource, [Parameter(Mandatory)][string]$Token, [datetime]$ExpiresUtc = ((Get-Date).ToUniversalTime().AddMinutes(50)), [string]$TenantId) $aud = Resolve-PimRestResource -Resource $Resource $tenant = Get-PimTenantId -TenantId $TenantId $cid = if ($global:PIM_ClientId) { $global:PIM_ClientId } elseif ($env:AZURE_CLIENT_ID) { $env:AZURE_CLIENT_ID } else { $null } if (-not $script:PimTokenCache) { $script:PimTokenCache = @{} } $script:PimTokenCache[("$aud|$tenant|$cid|interactive").ToLowerInvariant()] = [pscustomobject]@{ token = $Token; expiresUtc = $ExpiresUtc.ToUniversalTime() } } function Resolve-PimRestResource { param([Parameter(Mandatory)][string]$Resource) # 🪤 $script: IS NOT A CLOSURE FOR A DOT-SOURCED FUNCTION. It binds to the script scope # the function is CALLED from, so when another script dot-sources PIM-SqlStore (or the # downlink) WITHOUT this file, these tables are $null there and `.ContainsKey()` throws # "You cannot call a method on a null-valued expression". New-PimSqlConnection catches # that, tries the next credential, and ends up presenting NO credential -- Azure SQL then # answers "Login failed for user ''", which sends everyone looking at permissions. # Re-seeding here makes the alias table work from any scope; the value is a constant map, # so there is nothing to lose by rebuilding it. if (-not $script:PimRestResources) { $script:PimRestResources = Get-PimRestResourceMap } if ($script:PimRestResources.ContainsKey($Resource)) { return $script:PimRestResources[$Resource] } return ($Resource -replace '/+$','') # already a full audience URL } function ConvertTo-PimBase64Url { param([Parameter(Mandatory)][byte[]]$Bytes) [Convert]::ToBase64String($Bytes).TrimEnd('=').Replace('+','-').Replace('/','_') } function Get-PimTenantId { param([string]$TenantId) if ($TenantId) { return $TenantId } if ($global:PIM_TenantId) { return $global:PIM_TenantId } if ($env:PIM_TenantId) { return $env:PIM_TenantId } if ($env:AZURE_TENANT_ID) { return $env:AZURE_TENANT_ID } return $null } # ---- Managed Identity (App Service / IMDS) -------------------------------- function ConvertTo-PimTokenExpiry { param($ExpiresOn) # App Service IDENTITY_ENDPOINT -> unix seconds; older MSI/some -> a date string. try { if ("$ExpiresOn" -match '^\d+$') { return ([datetimeoffset]::FromUnixTimeSeconds([int64]$ExpiresOn)).UtcDateTime } } catch {} try { return ([datetimeoffset]"$ExpiresOn").UtcDateTime } catch {} return (Get-Date).ToUniversalTime().AddMinutes(50) } function Set-PimManagedIdentityHomeTenant { # LIC anti-copy (2026-10-03): the tid claim of a MANAGED IDENTITY token = the tenant this environment really runs in. # Recorded once per process in $global:PIM_ManagedIdentityTenantId; PIM-License.ps1 checks the licence against it, so a # licence cannot be made valid elsewhere by editing PIM_TenantId. Never throws. param([string]$Token) try { $seg = "$Token".Split('.')[1].Replace('-', '+').Replace('_', '/'); while ($seg.Length % 4) { $seg += '=' } $tid = "$(([Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($seg)) | ConvertFrom-Json).tid)".Trim().ToLowerInvariant() if ($tid -match '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$') { $global:PIM_ManagedIdentityTenantId = $tid } } catch { } } # ---- Azure Arc-enabled server (§100.63(b), the on-premises hybrid worker) --------------------------------------------- # The Arc agent (himds) serves the machine's system-assigned identity on http://localhost:40342 and sets IDENTITY_ENDPOINT # + IMDS_ENDPOINT machine-wide -- but NO IDENTITY_HEADER (that is App Service / Container Apps). Its token call is a # CHALLENGE: the first GET answers 401 with `WWW-Authenticate: Basic realm=`; the caller proves it may # use the identity by READING that file (only local Administrators and the local group "Hybrid agent extension # applications" can) and retries with `Authorization: Basic `. The file is single-use and short-lived. # 🔒 The realm is checked before anything is read: a .key file directly under %ProgramData%\AzureConnectedMachineAgent\Tokens, # at most 4 KB -- an endpoint that names any other file is refused (else a rogue local listener could make us send it the # content of an arbitrary file we can read). $script:PimArcDefaultEndpoint = 'http://localhost:40342/metadata/identity/oauth2/token' function Get-PimArcTokenDirectory { if ("$($global:PIM_ArcTokenDirectory)".Trim()) { return "$($global:PIM_ArcTokenDirectory)".Trim() } # offline test seam $pd = if ("$env:ProgramData".Trim()) { $env:ProgramData } else { 'C:\ProgramData' } return (Join-Path $pd 'AzureConnectedMachineAgent\Tokens') } function Get-PimArcIdentityEndpoint { # '' = not an Arc host. Arc = IDENTITY_ENDPOINT + IMDS_ENDPOINT set and no IDENTITY_HEADER / MSI_SECRET, or (env not yet # visible to this process -- a service started before the agent was installed) the himds service is present. if ($global:PIM_ArcIdentityEndpoint) { return "$($global:PIM_ArcIdentityEndpoint)" } # offline test seam if ("$env:IDENTITY_HEADER".Trim() -or "$env:MSI_SECRET".Trim()) { return '' } if ("$env:IDENTITY_ENDPOINT".Trim() -and "$env:IMDS_ENDPOINT".Trim()) { return "$env:IDENTITY_ENDPOINT".Trim() } if ($global:PIM_NoManagedIdentity -or $global:PIM_NoArcProbe) { return '' } if ($null -eq $script:PimArcHimdsPresent) { $script:PimArcHimdsPresent = $false if ("$env:OS" -eq 'Windows_NT') { try { $script:PimArcHimdsPresent = [bool](Get-Service -Name 'himds' -ErrorAction Stop) } catch { $script:PimArcHimdsPresent = $false } } } if ($script:PimArcHimdsPresent) { return $script:PimArcDefaultEndpoint } return '' } function Test-PimArcKeyFilePath { # PURE. Is this realm path a file the Arc agent would hand out? Directly under the token directory, *.key, no traversal. param([AllowEmptyString()][string]$Path, [Parameter(Mandatory)][string]$TokenDirectory) $p = "$Path".Trim().Trim('"') if (-not $p -or $p -match '\.\.' -or $p -notmatch '(?i)\.key$') { return $false } $full = $null; $dir = $null try { $full = [IO.Path]::GetFullPath($p); $dir = [IO.Path]::GetFullPath($TokenDirectory).TrimEnd('\', '/') } catch { $full = $null } if (-not $full -or -not $dir) { return $false } return ([IO.Path]::GetDirectoryName($full).TrimEnd('\', '/') -ieq $dir) } function Invoke-PimArcIdentityHttp { # ONE GET against the local Arc endpoint that never throws on an HTTP status: returns status, WWW-Authenticate, body. # No proxy (on-premises servers often have a system proxy; localhost must never go through it). Seam: PIM_ArcIdentityHttpStub. param([Parameter(Mandatory)][string]$Uri, [hashtable]$Headers = @{}, [int]$TimeoutSec = 30) if ($global:PIM_ArcIdentityHttpStub -is [scriptblock]) { return (& $global:PIM_ArcIdentityHttpStub $Uri $Headers) } $req = [System.Net.HttpWebRequest][System.Net.WebRequest]::Create($Uri) $req.Method = 'GET'; $req.Proxy = $null; $req.Timeout = $TimeoutSec * 1000; $req.ReadWriteTimeout = $TimeoutSec * 1000 foreach ($k in $Headers.Keys) { $req.Headers[$k] = "$($Headers[$k])" } $resp = $null try { $resp = $req.GetResponse() } catch [System.Net.WebException] { $resp = $_.Exception.Response; if (-not $resp) { throw } } catch { if ($_.Exception.InnerException -is [System.Net.WebException] -and $_.Exception.InnerException.Response) { $resp = $_.Exception.InnerException.Response } else { throw } } $out = $null try { $sr = New-Object System.IO.StreamReader($resp.GetResponseStream()) $body = $sr.ReadToEnd(); $sr.Close() $out = [pscustomobject]@{ status = [int]$resp.StatusCode; wwwAuthenticate = "$($resp.Headers['WWW-Authenticate'])"; body = $body } } finally { $resp.Close() } return $out } function Get-PimArcManagedIdentityToken { param([Parameter(Mandatory)][string]$Audience, [string]$Endpoint = '') $ep = if ("$Endpoint".Trim()) { "$Endpoint".Trim() } else { Get-PimArcIdentityEndpoint } if (-not $ep) { throw 'not an Azure Arc-enabled server (no IDENTITY_ENDPOINT/IMDS_ENDPOINT and no himds service)' } $u = "$($ep)?resource=$([uri]::EscapeDataString($Audience))&api-version=2020-06-01" $first = Invoke-PimArcIdentityHttp -Uri $u -Headers @{ Metadata = 'true' } $r = $first if ($first.status -eq 401) { $m = [regex]::Match("$($first.wwwAuthenticate)", '(?i)Basic\s+realm\s*=\s*"?([^"]+?)"?\s*$') if (-not $m.Success) { throw "Arc identity endpoint answered 401 without a Basic realm (WWW-Authenticate: '$($first.wwwAuthenticate)')" } $keyPath = $m.Groups[1].Value $dir = Get-PimArcTokenDirectory if (-not (Test-PimArcKeyFilePath -Path $keyPath -TokenDirectory $dir)) { throw "Arc identity endpoint named a key file outside '$dir' ('$keyPath') -- refusing to read it" } $len = -1; try { $len = (New-Object IO.FileInfo($keyPath)).Length } catch { $len = -1 } if ($len -gt 4096) { throw "Arc key file '$keyPath' is $len bytes (more than 4 KB) -- refusing" } $secret = $null; $readErr = $null try { $secret = [IO.File]::ReadAllText($keyPath).Trim() } catch { $readErr = $_.Exception } if ($readErr -is [System.UnauthorizedAccessException] -or ($readErr -and $readErr.InnerException -is [System.UnauthorizedAccessException])) { throw "cannot read the Arc key file '$keyPath' as $([Security.Principal.WindowsIdentity]::GetCurrent().Name) -- add this account to the local group 'Hybrid agent extension applications' (or run as a local administrator), then start a new process" } if ($readErr) { throw "cannot read the Arc key file '$keyPath': $($readErr.Message)" } if (-not $secret) { throw "the Arc key file '$keyPath' is empty" } $r = Invoke-PimArcIdentityHttp -Uri $u -Headers @{ Metadata = 'true'; Authorization = "Basic $secret" } $secret = $null } if ($r.status -ne 200) { throw "Arc identity endpoint answered HTTP $($r.status) for $Audience : $("$($r.body)".Substring(0, [Math]::Min(300, "$($r.body)".Length)))" } $j = "$($r.body)" | ConvertFrom-Json if (-not "$($j.access_token)") { throw "Arc identity endpoint returned no access_token for $Audience" } return [pscustomobject]@{ token = "$($j.access_token)"; expiresUtc = (ConvertTo-PimTokenExpiry $j.expires_on) } } function Test-PimMiTokenRetryable { <# PURE. MI-TOKEN-TRANSIENT: is a failed identity-endpoint call worth another try? No HTTP status (a timeout or a connection fault), 429 or 5xx = yes; any other status (400 identity not found, 401, 403, 404) = no. #> param([AllowNull()][object]$StatusCode) if ($null -eq $StatusCode -or "$StatusCode" -eq '' -or [int]$StatusCode -eq 0) { return $true } $s = [int]$StatusCode return ($s -eq 429 -or ($s -ge 500 -and $s -le 599)) } function Get-PimManagedIdentityToken { param([Parameter(Mandatory)][string]$Audience) $res = [uri]::EscapeDataString($Audience) # App Service / Functions (current): IDENTITY_ENDPOINT + IDENTITY_HEADER (api 2019-08-01) if ($env:IDENTITY_ENDPOINT -and $env:IDENTITY_HEADER) { $u = "$($env:IDENTITY_ENDPOINT)?resource=$res&api-version=2019-08-01" # 🔴 BUG-76 -- THIS BRANCH IGNORED THE CONFIGURED MI CLIENT ID, AND THE IMDS BRANCH BELOW DOES # NOT. That asymmetry is the whole defect: with a USER-ASSIGNED-ONLY identity there is nothing # for the endpoint to default to, so no token is issued and the caller ends up presenting no # credential at all. Azure SQL then answers "Login failed for user ''", which reads as a missing database user -- and it is not. # MEASURED on the greenfield slave 2026-08-27: ca-pim-tick carries `SystemAssigned, # UserAssigned` and logs "[mi] token via IDENTITY_ENDPOINT" happily, while ca-pim-downlink-s6 # carries UserAssigned ALONE and logs that line NOT AT ALL -- the absence was the clue. Its # engine apply failed every run on `Preflight FAILED: ... no SELECT 1`, which sent the search # to contained users and grants (where a user was duly created, and changed nothing). # Container Apps is exactly the host that attaches a user-assigned identity on its own. $miCid = if ($global:PIM_ManagedIdentityClientId) { "$($global:PIM_ManagedIdentityClientId)".Trim() } elseif ($env:PIM_ManagedIdentityClientId) { "$($env:PIM_ManagedIdentityClientId)".Trim() } else { '' } if ($miCid) { $u += "&client_id=$([uri]::EscapeDataString($miCid))" } # MI-TOKEN-TRANSIENT (SUP-2026-00323, internal rfa-sync 2026-10-27 02:16Z): ONE GET, no retry, so a single fault of the # endpoint failed the whole run while every other token of that window was issued. Retry a timeout / connection fault / # 429 / 5xx after a short wait; a 4xx (e.g. 400 "identity not found") is an answer, not a fault, and is thrown at once. # MANAGER-REFRESH-DEGRADES-CACHE (2026-10-27): on internal the endpoint was out for ~7 s twice in one day (02:16Z, # 20:00Z), longer than 2 s + 5 s -- a third try after 15 s covers that (22 s in all). $delays = if ($null -ne $global:PIM_MiTokenRetryDelaysSec) { @($global:PIM_MiTokenRetryDelaysSec) } else { @(2, 5, 15) } $r = $null for ($try = 0; ; $try++) { try { $r = Invoke-RestMethod -Method GET -Uri $u -Headers @{ 'X-IDENTITY-HEADER' = $env:IDENTITY_HEADER } -TimeoutSec 60; break } catch { $st = $null; try { if ($_.Exception.Response) { $st = [int]$_.Exception.Response.StatusCode } } catch {} if ($try -ge $delays.Count -or -not (Test-PimMiTokenRetryable -StatusCode $st)) { throw "the identity endpoint did not issue a token after $($try + 1) attempt(s): $($_.Exception.Message)" } try { [System.Console]::Out.WriteLine(" [mi] identity endpoint attempt $($try + 1) failed ($(if ($st) { "HTTP $st" } else { 'no answer' })) -- retrying in $($delays[$try]) s") } catch {} Start-Sleep -Seconds ([int]$delays[$try]) } } # §100.46: in the tick this is a start-up diagnostic (printed with the start-up block: on change / first of the day / PIM_LOG_VERBOSE=1) if ("$($r.access_token)") { $__mil = " [mi] token via IDENTITY_ENDPOINT (len $($r.access_token.Length))"; if ($global:PIM_LogQuiet -and (Get-Command Write-PimStartupLine -ErrorAction SilentlyContinue)) { Write-PimStartupLine -Message $__mil } else { try { [System.Console]::Out.WriteLine($__mil) } catch {} } } # Console.Out (not Write-Host): headless-safe from any scope (App Service has no console buffer; Write-Host throws there even from module scope) Set-PimManagedIdentityHomeTenant -Token "$($r.access_token)"; return [pscustomobject]@{ token = $r.access_token; expiresUtc = (ConvertTo-PimTokenExpiry $r.expires_on) } } # App Service (older / some Linux SKUs): MSI_ENDPOINT + MSI_SECRET (api 2017-09-01, header 'Secret') if ($env:MSI_ENDPOINT -and $env:MSI_SECRET) { $u = "$($env:MSI_ENDPOINT)?resource=$res&api-version=2017-09-01" $r = Invoke-RestMethod -Method GET -Uri $u -Headers @{ 'Secret' = $env:MSI_SECRET } -TimeoutSec 60 if ("$($r.access_token)") { try { [System.Console]::Out.WriteLine(" [mi] token via MSI_ENDPOINT (len $($r.access_token.Length))") } catch {} } # Console.Out: headless-safe from any scope Set-PimManagedIdentityHomeTenant -Token "$($r.access_token)"; return [pscustomobject]@{ token = $r.access_token; expiresUtc = (ConvertTo-PimTokenExpiry $r.expires_on) } } # Azure Arc-enabled server (on-premises hybrid worker, §100.63(b)): IDENTITY_ENDPOINT without IDENTITY_HEADER, 401 challenge if (Get-PimArcIdentityEndpoint) { $a = Get-PimArcManagedIdentityToken -Audience $Audience try { [System.Console]::Out.WriteLine(" [mi] token via the Azure Arc agent (len $($a.token.Length))") } catch {} Set-PimManagedIdentityHomeTenant -Token "$($a.token)"; return $a } # IMDS (Azure VM) $u = "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=$res" if ($global:PIM_ManagedIdentityClientId) { $u += "&client_id=$($global:PIM_ManagedIdentityClientId)" } $r = Invoke-RestMethod -Method GET -Uri $u -Headers @{ Metadata = 'true' } -TimeoutSec 5 Set-PimManagedIdentityHomeTenant -Token "$($r.access_token)"; return [pscustomobject]@{ token = $r.access_token; expiresUtc = (ConvertTo-PimTokenExpiry $r.expires_on) } } # ---- SPN client secret ---------------------------------------------------- function Get-PimClientSecretToken { param([Parameter(Mandatory)][string]$TenantId,[Parameter(Mandatory)][string]$ClientId,[Parameter(Mandatory)][string]$ClientSecret,[Parameter(Mandatory)][string]$Audience) $body = @{ grant_type='client_credentials'; client_id=$ClientId; client_secret=$ClientSecret; scope="$Audience/.default" } $r = Invoke-RestMethod -Method POST -Uri "https://login.microsoftonline.com/$TenantId/oauth2/v2.0/token" -ContentType 'application/x-www-form-urlencoded' -Body $body -TimeoutSec 60 return [pscustomobject]@{ token = $r.access_token; expiresUtc = (Get-Date).ToUniversalTime().AddSeconds([int]$r.expires_in - 60) } } # ---- SPN certificate (signed JWT assertion) ------------------------------- function Get-PimClientCertToken { param([Parameter(Mandatory)][string]$TenantId,[Parameter(Mandatory)][string]$ClientId,[Parameter(Mandatory)][System.Security.Cryptography.X509Certificates.X509Certificate2]$Certificate,[Parameter(Mandatory)][string]$Audience) $tokenUrl = "https://login.microsoftonline.com/$TenantId/oauth2/v2.0/token" $now = [DateTimeOffset]::UtcNow $x5t = ConvertTo-PimBase64Url -Bytes $Certificate.GetCertHash() # SHA1 thumbprint bytes $header = @{ alg='RS256'; typ='JWT'; x5t=$x5t } | ConvertTo-Json -Compress $claims = @{ aud=$tokenUrl; iss=$ClientId; sub=$ClientId; jti=([guid]::NewGuid().ToString()) nbf=$now.ToUnixTimeSeconds(); exp=$now.AddMinutes(10).ToUnixTimeSeconds() } | ConvertTo-Json -Compress $h = ConvertTo-PimBase64Url -Bytes ([Text.Encoding]::UTF8.GetBytes($header)) $c = ConvertTo-PimBase64Url -Bytes ([Text.Encoding]::UTF8.GetBytes($claims)) $unsigned = "$h.$c" $rsa = [System.Security.Cryptography.X509Certificates.RSACertificateExtensions]::GetRSAPrivateKey($Certificate) if (-not $rsa) { throw "Certificate has no usable RSA private key for JWT signing." } $sigBytes = $rsa.SignData([Text.Encoding]::UTF8.GetBytes($unsigned), [System.Security.Cryptography.HashAlgorithmName]::SHA256, [System.Security.Cryptography.RSASignaturePadding]::Pkcs1) $jwt = "$unsigned." + (ConvertTo-PimBase64Url -Bytes $sigBytes) $body = @{ grant_type='client_credentials'; client_id=$ClientId; scope="$Audience/.default" client_assertion_type='urn:ietf:params:oauth:client-assertion-type:jwt-bearer'; client_assertion=$jwt } $r = Invoke-RestMethod -Method POST -Uri $tokenUrl -ContentType 'application/x-www-form-urlencoded' -Body $body -TimeoutSec 60 return [pscustomobject]@{ token = $r.access_token; expiresUtc = (Get-Date).ToUniversalTime().AddSeconds([int]$r.expires_in - 60) } } function Get-PimTokenTenantId { <# SEC-12 -- read the tenant a JWT actually belongs to, from its own claims. 🔑 ASKING FOR A TENANT IS NOT THE SAME AS BEING GIVEN ONE, and until this existed there was no way to tell the difference: a token for the wrong directory is a perfectly valid string that succeeds at the call site and fails somewhere else as a permissions error. One `Split('.')[1]` turns that into a fact. PURE: no network, no validation of the signature (that is the resource's job) -- this only reads what the token SAYS about itself, which is exactly what is needed to catch "you asked for A and something handed you B". Returns the lowercased `tid`, or '' when the string is not a readable JWT -- never throws, because a caller reaching for a sanity check must not be broken BY the sanity check. #> [CmdletBinding()] param([string]$Token) try { $parts = "$Token".Split('.') if ($parts.Count -lt 2) { return '' } $p = $parts[1].Replace('-', '+').Replace('_', '/') while ($p.Length % 4) { $p += '=' } $claims = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($p)) | ConvertFrom-Json return "$($claims.tid)".Trim().ToLowerInvariant() } catch { return '' } } function Get-PimTokenAppId { <# SEC-12 companion: the appid a JWT claims. Same contract as Get-PimTokenTenantId. #> [CmdletBinding()] param([string]$Token) try { $parts = "$Token".Split('.') if ($parts.Count -lt 2) { return '' } $p = $parts[1].Replace('-', '+').Replace('_', '/') while ($p.Length % 4) { $p += '=' } $claims = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($p)) | ConvertFrom-Json return "$($claims.appid)".Trim().ToLowerInvariant() } catch { return '' } } function Resolve-PimCertificate { param([string]$Thumbprint) $tp = if ($Thumbprint) { $Thumbprint } elseif ($global:PIM_CertThumbprint) { $global:PIM_CertThumbprint } else { $null } if (-not $tp) { return $null } $tp = ($tp -replace '\s','').ToUpperInvariant() foreach ($store in @('Cert:\CurrentUser\My','Cert:\LocalMachine\My')) { $c = Get-ChildItem $store -ErrorAction SilentlyContinue | Where-Object { $_.Thumbprint -eq $tp } | Select-Object -First 1 if ($c) { return $c } } return $null } # ---- interactive (delegated) -- BREAK-GLASS / emergency on a client PC ---- # Dependency-free auth-code + PKCE loopback. No MSAL, no Graph/Az modules. # Used when there is no MI and no SPN credential (a cloud-only admin PC running # the emergency edition): the operator signs in as THEMSELVES, so the resulting # SQL/Graph/ARM action is audited under the human identity, not a shared app. # Returns a raw access token for $Audience. Edge is launched explicitly to avoid # the system-default-browser state-mismatch bug; any first-party public client # accepts an arbitrary localhost redirect port. function Get-PimInteractiveClientId { <# PURE. The first-party public client for a browser sign-in to -Audience: Graph -> the Microsoft Graph CLI client (pre-authorised for Graph only); everything else (ARM, Azure SQL, Key Vault, Storage) -> the Microsoft Azure CLI client APP ID (an app registration id used over REST, not the az program). AADSTS650057 on ARM, 2026-10-10. #> param([string]$Audience) # ONE client for every audience (2026-10-10, a customer's install stalled on a hidden SECOND sign-in window when Graph # used a different client): the Azure CLI first-party public client is pre-authorised for Graph AND ARM / SQL / Key # Vault / Storage in every tenant, so ONE browser sign-in serves all of them through its refresh token. return '04b07795-8ddb-461a-bbee-02f9e1bf7b46' } # REQ 100.76 (2026-10-11, a new customer's hosting step: "Login failed for user ''"): a signed-in # build hands its step processes the person's REFRESH token(s) for the target tenant in $env:PIM_HANDOFF_REFRESH # ({ "|": "" }, that process's environment only), instead of a pre-minted Azure SQL # ACCESS token. The step creates the SQL server, sets its Entra admin and adds the person to the SQL admin group AFTER the # build minted, so only a token minted at connect time carries that right. Entries for another tenant are never taken. function Import-PimHandoffRefresh { param([string]$TenantId, [string]$Json = "$env:PIM_HANDOFF_REFRESH") if (-not "$Json".Trim()) { return 0 } if (-not ($global:PimInteractiveRefresh -is [hashtable])) { $global:PimInteractiveRefresh = @{} } $want = "$TenantId".Trim().ToLowerInvariant(); $n = 0 try { $o = "$Json" | ConvertFrom-Json } catch { Write-Verbose "PIM_HANDOFF_REFRESH unreadable: $($_.Exception.Message)"; return 0 } foreach ($p in @($o.PSObject.Properties)) { $k = "$($p.Name)".Trim().ToLowerInvariant(); $v = "$($p.Value)".Trim() if (-not $v -or $k -notmatch '^[^|]+\|[^|]+$') { continue } if ($want -and $k.Split('|')[0] -ne $want) { continue } if (-not $global:PimInteractiveRefresh.ContainsKey($k)) { $global:PimInteractiveRefresh[$k] = $v; $n++ } } return $n } # REQ 100.76: the refresh tokens a signed-in build hands a step for -TenantId, as the JSON for $env:PIM_HANDOFF_REFRESH # ('' when there are none). Pure over -Map (default: this process's in-memory refresh map). function Get-PimHandoffRefreshJson { param([Parameter(Mandatory)][string]$TenantId, [object]$Map = $global:PimInteractiveRefresh) if (-not ($Map -is [hashtable]) -or -not $Map.Count) { return '' } $want = "$TenantId".Trim().ToLowerInvariant(); $out = [ordered]@{} foreach ($k in @($Map.Keys)) { $kl = "$k".ToLowerInvariant(); if ($kl.Split('|')[0] -eq $want -and "$($Map[$k])".Trim()) { $out[$kl] = "$($Map[$k])" } } if (-not $out.Count) { return '' } return ($out | ConvertTo-Json -Compress) } function New-PimLoopbackListeners { # The browser sign-in's redirect listeners: the IPv4 loopback on a free port AND the IPv6 loopback on the SAME port when # the machine has IPv6 (a "localhost" redirect may resolve to either). Loopback only. A port the IPv6 side cannot take # is retried on a new one (a few times); no IPv6 at all = the IPv4 listener alone, as before. for ($try = 0; $try -lt 5; $try++) { $v4 = New-Object System.Net.Sockets.TcpListener([System.Net.IPAddress]::Loopback, 0) $v4.Start() if (-not [System.Net.Sockets.Socket]::OSSupportsIPv6) { return $v4 } $p = ([System.Net.IPEndPoint]$v4.LocalEndpoint).Port try { $v6 = New-Object System.Net.Sockets.TcpListener([System.Net.IPAddress]::IPv6Loopback, $p) $v6.Start() return @($v4, $v6) # the caller wraps it in @(): one flat list } catch { $v4.Stop() } } $only = New-Object System.Net.Sockets.TcpListener([System.Net.IPAddress]::Loopback, 0); $only.Start() return $only } function Get-PimInteractiveToken { param([Parameter(Mandatory)][string]$Audience,[string]$TenantId,[string]$ClientId, # section 9 account sign-in clarity: force a brand-new credential prompt (prompt=login) # instead of reusing a cached account. -ExpectedAccount, if the cached account # differs, also forces a fresh prompt so a stale account is never used silently. [switch]$ForceFreshAccount,[string]$ExpectedAccount, # REQ 100.76: only the silent refresh-token exchange -- $null instead of a browser window when there is none. [switch]$RefreshOnly) $tenant = if ($TenantId) { $TenantId } elseif (Get-PimTenantId) { Get-PimTenantId } else { 'organizations' } # Default public client PER AUDIENCE (2026-10-10, a customer's managed install failed with AADSTS650057 "Invalid # resource" for ARM): the Microsoft Graph CLI client (14d82eec-...) is pre-authorised for GRAPH only. Azure Resource # Manager, Azure SQL, Key Vault and Storage use the Microsoft Azure CLI first-party public client APP ID # (04b07795-..., an app registration id used over REST -- not the az program), pre-authorised for those in every tenant. $cid = if ($ClientId) { $ClientId } elseif ($global:PIM_InteractiveClientId) { $global:PIM_InteractiveClientId } else { Get-PimInteractiveClientId -Audience $Audience } # ONE sign-in: a refresh token from an earlier browser sign-in (same tenant + client) is exchanged for this audience -- # no second window. Falls through to the browser only when there is none or the exchange is refused. if (-not $global:PimInteractiveRefresh) { $global:PimInteractiveRefresh = @{} } $rtKey = "$tenant|$cid".ToLowerInvariant() if (-not $global:PimInteractiveRefresh.ContainsKey($rtKey)) { [void](Import-PimHandoffRefresh -TenantId $tenant) } if ($global:PimInteractiveRefresh.ContainsKey($rtKey) -and -not $ForceFreshAccount) { try { $rr = Invoke-RestMethod -Method POST -Uri "https://login.microsoftonline.com/$tenant/oauth2/v2.0/token" -ContentType 'application/x-www-form-urlencoded' -Body @{ client_id = $cid; grant_type = 'refresh_token'; refresh_token = $global:PimInteractiveRefresh[$rtKey] scope = "$Audience/.default offline_access openid profile" } if ($rr.refresh_token) { $global:PimInteractiveRefresh[$rtKey] = $rr.refresh_token } if ($rr.access_token) { return [pscustomobject]@{ token = $rr.access_token; expiresUtc = (Get-Date).ToUniversalTime().AddSeconds([int]$rr.expires_in - 60) } } } catch { Write-Verbose "refresh-token exchange for $Audience refused: $($_.Exception.Message) -- browser sign-in" } } if ($RefreshOnly) { return $null } $bytes = New-Object byte[] 32 [System.Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($bytes) $verifier = ConvertTo-PimBase64Url -Bytes $bytes $sha = [System.Security.Cryptography.SHA256]::Create() $challenge = ConvertTo-PimBase64Url -Bytes ($sha.ComputeHash([System.Text.Encoding]::ASCII.GetBytes($verifier))) $state = [guid]::NewGuid().ToString('N') # The redirect says "localhost" (what Entra's public clients accept on any port), and a browser may resolve it to the IPv4 # OR the IPv6 loopback: listen on BOTH, on the same port (a customer 2026-10-21: an IPv4-only listener, the jump host's browser # went to [::1] -> "localhost refused to connect"). Loopback only -- never another interface. $tcps = @(New-PimLoopbackListeners) $port = ([System.Net.IPEndPoint]$tcps[0].LocalEndpoint).Port $redirect = "http://localhost:$port/" # section 9 prompt clarity: select_account (always show the picker so a stale cached account # is never reused silently) unless a fresh sign-in is forced -> login. ConvertTo- # PimAuthCodePrompt is in PIM-AuthDiagnostics; fall back if it isn't dot-sourced. if (Get-Command ConvertTo-PimAuthCodePrompt -ErrorAction SilentlyContinue) { $prompt = ConvertTo-PimAuthCodePrompt -ForceFresh:$ForceFreshAccount -KnownStaleAccount $ExpectedAccount } else { $prompt = if ($ForceFreshAccount) { 'login' } else { 'select_account' } } $scope = "$Audience/.default offline_access openid profile" $authUrl = "https://login.microsoftonline.com/$tenant/oauth2/v2.0/authorize" + "?client_id=$cid&response_type=code&response_mode=query" + "&redirect_uri=$([uri]::EscapeDataString($redirect))" + "&scope=$([uri]::EscapeDataString($scope))&state=$state" + "&code_challenge=$challenge&code_challenge_method=S256&prompt=$prompt" if ("$ExpectedAccount".Trim()) { $authUrl += "&login_hint=$([uri]::EscapeDataString($ExpectedAccount.Trim()))" } $edge = @( (Join-Path ${env:ProgramFiles(x86)} 'Microsoft\Edge\Application\msedge.exe'), (Join-Path $env:ProgramFiles 'Microsoft\Edge\Application\msedge.exe') ) | Where-Object { $_ -and (Test-Path $_) } | Select-Object -First 1 Write-Host " [interactive] sign-in required for $Audience (loopback $redirect)" -ForegroundColor Yellow # 2026-10-10: tell the installer's progress feed BEFORE a browser window opens (a hidden window waited silently and the # customer saw "nothing happening"); the installer registers $global:PIM_SignInNotify. if ($global:PIM_SignInNotify -is [scriptblock]) { try { & $global:PIM_SignInNotify $Audience $tenant } catch { } } # A person whose browser opens in the wrong profile can paste the address into the right one (2026-10-10). Write-Host " If the browser opens in the wrong profile or account, open this address in the browser profile of an admin of tenant ${tenant}:" -ForegroundColor Yellow Write-Host " $authUrl" -ForegroundColor DarkGray # 2026-10-10 (a customer's install): a normal Edge window opens the DEFAULT profile, whose saved account signs in silently # -- the person never got a credential prompt and landed in the wrong tenant. An InPrivate window has no saved account, # so the sign-in form always appears. PIM_SIGNIN_BROWSER=none: open nothing, the person pastes the address above. $mode = "$env:PIM_SIGNIN_BROWSER".Trim().ToLowerInvariant() if ($mode -eq 'none') { Write-Host ' (PIM_SIGNIN_BROWSER=none: no browser opened -- paste the address above into the right browser)' -ForegroundColor Yellow } elseif ($edge) { Start-Process -FilePath $edge -ArgumentList @('--inprivate', '--new-window', $authUrl) -WhatIf:$false } else { Start-Process $authUrl -WhatIf:$false } # fall back to default browser if Edge absent $query = $null try { # SIGNIN-PAGE (owner 2026-10-31, Invardia's Wait-AuthCode fix): only the answer (code / error) ends the wait, every # request gets a page, and the page keeps answering 10 s after it (engine/_shared/PIM-LoopbackAnswer.ps1). if (-not (Get-Command Wait-PimLoopbackAnswer -ErrorAction SilentlyContinue)) { # ===== BEGIN inlined helper: PIM-LoopbackAnswer.ps1 ===== # PIM-LoopbackAnswer.ps1 -- the ONE wait for a browser sign-in's answer on the localhost loopback (auth code + PKCE). # Used by every PIM browser sign-in: the REST library's own (signed-in setup scripts), the mail setup (Initialize-PimMailSender, # Set-PimSmtpRelayPassword, Set-PimSqlTier), the engine-permissions grant and the Activator deploy scripts. The tools load the # identical copy in tools\setup (they may not load engine files). Build-PimSupportScripts inlines it into each published script. # # Owner 2026-10-30/31 ("people are reporting this error" -- "This site can't be reached"; "yes fix the sign-in page in all # scripts"), the same fix as Invardia's Wait-AuthCode: the old loops took the FIRST request -- often the browser's preload # of the address or its icon request -- and a request that never sent a line held the loop, and once the answer came the # listener stopped at once, so the browser's real load or a reload got "This site can't be reached". Now: # 1. only a request whose query carries code= or error= is the answer; the loop goes on until one comes; # 2. EVERY request gets a page (waiting / signed in / failed), never a closed connection; # 3. after the answer the listener keeps answering for -GraceSeconds (10), so a second load shows "Signed in" too; # 4. it throws only when no answer came within -Seconds. # A connection that sends nothing (a speculative preconnect) is dropped after -ReadTimeoutMs and never blocks the loop. function Test-PimLoopbackAnswerQuery { # PURE. Is this request line the sign-in's answer? Returns the query string ('' when not). param([AllowEmptyString()][AllowNull()][string]$RequestLine) $m = [regex]::Match("$RequestLine", '^GET /[^?\s]*\?(?\S+) HTTP') if (-not $m.Success) { return '' } $q = $m.Groups['q'].Value if ($q -match '(^|&)(code|error)=') { return $q } return '' } function Get-PimLoopbackPage { # PURE. The page for one request: before the answer 'waiting', after it 'done' (or 'failed' when the answer is an error). param([ValidateSet('waiting', 'done', 'failed')][string]$Kind, [string]$ReturnTo = 'PowerShell', [string]$ErrorText = '') $body = switch ($Kind) { 'waiting' { '

Waiting for the sign-in...

Finish the sign-in in the other tab.' } 'done' { "

Sign-in complete.

You can close this tab and return to $([System.Net.WebUtility]::HtmlEncode($ReturnTo))." } 'failed' { "

Sign-in failed

$([System.Net.WebUtility]::HtmlEncode($ErrorText))
Return to $([System.Net.WebUtility]::HtmlEncode($ReturnTo))." } } "$body" } function Wait-PimLoopbackAnswer { <# Waits on the loopback listener(s) (TcpListener, started by the caller; both loopbacks may share one port) for the sign-in's answer and returns its raw query string. Throws only when no answer came within -Seconds. The caller still checks state / error / code and stops the listeners (this function does not stop them). #> param([Parameter(Mandatory)][object[]]$Listeners, [int]$Seconds = 300, [int]$GraceSeconds = 10, [int]$ReadTimeoutMs = 2000, [string]$ReturnTo = 'PowerShell', [string]$TimeoutMessage = '') $deadline = [DateTime]::UtcNow.AddSeconds($Seconds); $graceEnd = $null $answer = $null; $kind = 'waiting'; $errText = '' while ($true) { $now = [DateTime]::UtcNow if ($answer) { if ($now -ge $graceEnd) { break } } elseif ($now -ge $deadline) { break } $ready = @($Listeners | Where-Object { $_.Pending() })[0] if (-not $ready) { Start-Sleep -Milliseconds 100; continue } $client = $null try { $client = $ready.AcceptTcpClient() $client.ReceiveTimeout = $ReadTimeoutMs $stream = $client.GetStream() $reader = New-Object System.IO.StreamReader($stream) $line = $null try { $line = $reader.ReadLine() } catch { $line = $null } # a connection that sends nothing: dropped, never waited on if ($null -eq $line) { continue } if (-not $answer) { $q = Test-PimLoopbackAnswerQuery -RequestLine $line if ($q) { $answer = $q; $graceEnd = [DateTime]::UtcNow.AddSeconds($GraceSeconds) if ($q -match '(^|&)error=') { $kind = 'failed' $d = [regex]::Match($q, '(^|&)error_description=(?[^&]*)') $errText = if ($d.Success) { [uri]::UnescapeDataString(($d.Groups['d'].Value -replace '\+', ' ')) } else { '' } } else { $kind = 'done' } } } $html = Get-PimLoopbackPage -Kind $kind -ReturnTo $ReturnTo -ErrorText $errText $bytes = [Text.Encoding]::UTF8.GetBytes($html) $head = [Text.Encoding]::ASCII.GetBytes("HTTP/1.1 200 OK`r`nContent-Type: text/html; charset=utf-8`r`nContent-Length: $($bytes.Length)`r`nConnection: close`r`n`r`n") $stream.Write($head, 0, $head.Length); $stream.Write($bytes, 0, $bytes.Length); $stream.Flush() } catch { } # a browser that closed the connection itself: nothing to answer finally { if ($client) { try { $client.Close() } catch { } } } } if (-not $answer) { throw $(if ("$TimeoutMessage".Trim()) { $TimeoutMessage } else { "Timed out ($([int]($Seconds / 60)) minutes) waiting for the browser to finish the sign-in." }) } return $answer } # ===== END inlined helper: PIM-LoopbackAnswer.ps1 ===== } $query = Wait-PimLoopbackAnswer -Listeners $tcps -Seconds 300 -TimeoutMessage 'Timed out (5 min) waiting for the sign-in to come back to this window -- nothing was changed. Run the same command again and finish the sign-in within 5 minutes.' } finally { foreach ($t in $tcps) { try { $t.Stop() } catch { } } } $kv = @{} foreach ($pair in ($query -split '&')) { $k,$v = $pair -split '=',2; $kv[$k] = if ($null -ne $v) { [uri]::UnescapeDataString(($v -replace '\+',' ')) } else { '' } } if ($kv['error']) { throw "Sign-in failed: $($kv['error']) -- $($kv['error_description'])" } if ($kv['state'] -ne $state) { throw 'State mismatch on the loopback redirect -- close ALL browser windows and retry.' } if (-not $kv['code']) { throw 'Sign-in redirect carried no authorization code.' } $r = Invoke-RestMethod -Method POST -Uri "https://login.microsoftonline.com/$tenant/oauth2/v2.0/token" -ContentType 'application/x-www-form-urlencoded' -Body @{ client_id = $cid grant_type = 'authorization_code' code = $kv['code'] redirect_uri = $redirect code_verifier = $verifier scope = $scope } # The account must belong to the TARGET tenant: a token from another tenant (wrong browser profile / account) stops here, # before any call is made with it. if ($tenant -match '^[0-9a-fA-F-]{36}$') { $tid = '' try { $pp = "$($r.access_token)".Split('.')[1].Replace('-', '+').Replace('_', '/'); while ($pp.Length % 4) { $pp += '=' } $tid = "$(([Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($pp)) | ConvertFrom-Json).tid)".Trim() } catch { $tid = '' } if ($tid -and $tid -ne $tenant) { throw "You signed in to tenant $tid, but this is for tenant $tenant -- sign in with an account of $tenant (use that admin's browser profile)." } } if ($r.refresh_token) { $global:PimInteractiveRefresh["$tenant|$cid".ToLowerInvariant()] = $r.refresh_token } # in memory only, never written return [pscustomobject]@{ token = $r.access_token; expiresUtc = (Get-Date).ToUniversalTime().AddSeconds([int]$r.expires_in - 60) } } # ---- framework 8.13 INSTALL-HANDOFF: one sign-in across Invardia's bootstrap and the product installer ---- # Invardia's bootstrap (Install-Invardia) signs the person in ONCE (browser PKCE, or the Cloud Shell user token) and, in # the SAME process, just before it calls Install-PimManager, sets $env:PIM_HANDOFF_TOKENS = # { "https://management.azure.com/": { "access_token": "...", "expires_on": }, "https://graph.microsoft.com/": {...}, # "https://database.windows.net/": {...} } # ACCESS tokens only (no refresh token crosses the boundary); an audience it could not get is simply absent; it removes # the variable in a finally block. Install-PimManager reads it ONCE (Import-PimHandoffTokens), clears it, and keeps only # the tokens that pass the checks below in $global:PimHandoffTokens -- memory only, never a file, log, transcript or event. # Every check is on the token's OWN claims (claims read only: this is our own process boundary, and the resource still # validates the signature on every call). # tid == the installation's tenant (config.tenantId). A mismatch on ANY audience refuses the WHOLE handoff: the # bootstrap signed in to a different tenant than the answers name, so none of it is trusted. # a USER the token must be DELEGATED: Entra puts 'scp' only in delegated (user) tokens and 'roles' only (no scp) in # application tokens; 'idtyp' is an OPTIONAL claim, so 'idtyp = app' refuses but its absence proves nothing. # Rule: idtyp is not 'app' AND scp is present AND oid is a GUID. ('roles' beside scp is allowed: a user can # hold app roles.) # expiry the earlier of expires_on and the JWT's exp must be more than 5 minutes away. # A token failing a check (other than the tenant) is refused for ITS audience only; that audience falls back to PIM's own # browser sign-in (Get-PimInteractiveToken, pinned to the tenant) when it is first needed, and that sign-in's in-memory # refresh token then covers the other audiences PIM needs later. function Get-PimHandoffAudienceKey { # PURE. The comparison key of an audience: 'https://management.azure.com/' and 'https://management.azure.com' are one. param([string]$Audience) return "$Audience".Trim().TrimEnd('/').ToLowerInvariant() } function ConvertFrom-PimHandoffJwt { # PURE. The payload claims of a JWT, or $null for anything that is not one. Never throws; never returns the token. param([string]$Token) try { $parts = "$Token".Split('.') if ($parts.Count -ne 3 -or -not $parts[1]) { return $null } $p = $parts[1].Replace('-', '+').Replace('_', '/') while ($p.Length % 4) { $p += '=' } $c = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($p)) | ConvertFrom-Json -ErrorAction Stop if ($null -eq $c -or $c -is [string] -or $c -is [ValueType]) { return $null } return $c } catch { return $null } } function ConvertTo-PimHandoffExpiry { # PURE. expires_on as a bootstrap hands it over -- epoch seconds (number or digit string: the MSAL / managed-identity / # Cloud Shell shape) or a date -- as a UTC [datetime], or $null when it cannot be read. param($Value) if ($null -eq $Value) { return $null } if ($Value -is [datetime]) { return $Value.ToUniversalTime() } if ($Value -is [DateTimeOffset]) { return $Value.UtcDateTime } $s = "$Value".Trim() if (-not $s) { return $null } if ($s -match '^\d{9,11}(\.\d+)?$') { try { return [DateTimeOffset]::FromUnixTimeSeconds([long][math]::Floor([double]$s)).UtcDateTime } catch { return $null } } $d = [DateTimeOffset]::MinValue if ([DateTimeOffset]::TryParse($s, [Globalization.CultureInfo]::InvariantCulture, [Globalization.DateTimeStyles]::AssumeUniversal, [ref]$d)) { return $d.UtcDateTime } return $null } function Test-PimHandoffToken { <# PURE given -NowUtc. Check ONE handed-over access token (see the block above). Returns @{ ok; tenantMismatch; reason; expiresUtc; userName }. The reason never contains token text. #> param([string]$Token, $ExpiresOn, [Parameter(Mandatory)][string]$TenantId, [datetime]$NowUtc = [datetime]::UtcNow, [int]$MinValidityMinutes = 5, # APP-IDENTITY INSTALL (owner via Invardia 2026-10-21, Install-Invardia -ViaSupportApp): an APPLICATION token is # accepted ONLY when its appid is exactly this app -- the Invardia Support app of the session open in this process. # Empty (the default) = the user rule below, unchanged. [string]$AllowedAppId = '') $no = { param($why, [bool]$mismatch = $false) @{ ok = $false; tenantMismatch = $mismatch; reason = $why; expiresUtc = $null; userName = ''; kind = '' } } if (-not "$Token".Trim()) { return (& $no 'no access token was handed over') } $c = ConvertFrom-PimHandoffJwt -Token $Token if (-not $c) { return (& $no 'it is not a readable access token') } $want = "$TenantId".Trim().ToLowerInvariant() $tid = "$($c.tid)".Trim().ToLowerInvariant() if (-not $tid) { return (& $no 'it names no tenant (no tid claim)') } if ($tid -ne $want) { return (& $no "it is for tenant $tid, not this installation's tenant $want" $true) } $idtyp = "$($c.idtyp)".Trim().ToLowerInvariant() $kind = 'user' if ($idtyp -eq 'app' -or -not "$($c.scp)".Trim()) { $appid = "$(if ($c.appid) { $c.appid } else { $c.azp })".Trim().ToLowerInvariant() if (-not "$AllowedAppId".Trim() -or $appid -ne "$AllowedAppId".Trim().ToLowerInvariant()) { return (& $no 'it is an application token, not a signed-in user''s (idtyp=app or no delegated scp claim)') } $kind = 'app' } if ("$($c.oid)".Trim() -notmatch '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$') { return (& $no $(if ($kind -eq 'app') { 'it carries no object id (oid) of the application' } else { 'it carries no user object id (oid)' })) } $given = ConvertTo-PimHandoffExpiry -Value $ExpiresOn $jwtExp = $null if ("$($c.exp)".Trim() -match '^\d+$') { try { $jwtExp = [DateTimeOffset]::FromUnixTimeSeconds([long]"$($c.exp)".Trim()).UtcDateTime } catch { $jwtExp = $null } } $eff = @(@($given, $jwtExp) | Where-Object { $_ } | Sort-Object) | Select-Object -First 1 if (-not $eff) { return (& $no 'it carries no expiry (no expires_on and no exp claim)') } if ($eff -le $NowUtc.ToUniversalTime().AddMinutes($MinValidityMinutes)) { return (& $no ("it expires at {0:yyyy-MM-dd HH:mm} UTC -- less than {1} minutes left" -f $eff, $MinValidityMinutes)) } $upn = if ($kind -eq 'app') { '' } else { "$(if ($c.upn) { $c.upn } elseif ($c.unique_name) { $c.unique_name } elseif ($c.preferred_username) { $c.preferred_username } else { '' })".Trim() } return @{ ok = $true; tenantMismatch = $false; reason = ''; expiresUtc = $eff; userName = $upn; kind = $kind } } function Import-PimHandoffTokens { <# PURE given -NowUtc. Read the bootstrap's PIM_HANDOFF_TOKENS JSON and check every audience against -TenantId. Returns @{ present; refused; reason; accepted = @{ = @{ token; expiresUtc } }; fallback = @(@{ audience; reason }) }. refused = a token for ANOTHER tenant was handed over: NOTHING is accepted (reason names the audience and both tenants). fallback = audiences that were handed over but refused (each falls back to the browser sign-in when it is needed). The values in 'accepted' are the only place the tokens go; the caller keeps them in memory and never prints them. #> param([string]$Json, [Parameter(Mandatory)][string]$TenantId, [datetime]$NowUtc = [datetime]::UtcNow, # APP-IDENTITY INSTALL: the Invardia Support app of the session open in this process ('' = users only, as before) [string]$AllowedAppId = '') $out = @{ present = $false; refused = $false; reason = ''; accepted = @{}; fallback = @(); kind = '' } if (-not "$Json".Trim()) { return $out } $out.present = $true $o = $null try { $o = $Json | ConvertFrom-Json -ErrorAction Stop } catch { $o = $null } if ($null -eq $o -or $o -is [string] -or $o -is [ValueType] -or $o -is [System.Array]) { $out.fallback = @(@{ audience = '(all)'; reason = 'the handed-over sign-in could not be read (not a JSON object)' }) return $out } $mismatch = @() foreach ($p in $o.PSObject.Properties) { $v = $p.Value $tok = ''; $exp = $null if ($v -is [string]) { $tok = $v } elseif ($null -ne $v -and $v.PSObject.Properties['access_token']) { $tok = "$($v.access_token)"; if ($v.PSObject.Properties['expires_on']) { $exp = $v.expires_on } } $t = Test-PimHandoffToken -Token $tok -ExpiresOn $exp -TenantId $TenantId -NowUtc $NowUtc -AllowedAppId $AllowedAppId if ($t.tenantMismatch) { $mismatch += "$($p.Name): $($t.reason)" } elseif ($t.ok) { $out.accepted[(Get-PimHandoffAudienceKey $p.Name)] = @{ token = $tok; expiresUtc = $t.expiresUtc; userName = $t.userName; kind = $t.kind } } else { $out.fallback += @{ audience = "$($p.Name)"; reason = $t.reason } } $tok = $null } # One principal per handoff: a person's tokens beside an application's would make "who did this" a guess -- refused whole. $kinds = @($out.accepted.Values | ForEach-Object { "$($_.kind)" } | Select-Object -Unique) if ($kinds.Count -gt 1) { $out.refused = $true; $out.accepted = @{} $out.reason = 'the sign-in handed over by Invardia''s installer is REFUSED as a whole -- it mixes a person''s tokens with an application''s; nothing from it is used.' return $out } $out.kind = "$(@($kinds)[0])" if ($mismatch.Count) { $out.refused = $true $out.accepted = @{} $out.reason = ('the sign-in handed over by Invardia''s installer is REFUSED as a whole -- ' + ($mismatch -join '; ') + '. It signed in to a different tenant than the installation answers name; nothing from it is used.') } return $out } function Get-PimHandoffToken { <# The validated handed-over access token for -Audience in -TenantId ($global:PimHandoffTokens, memory only), or '' when there is none, it is for another tenant, or it has under 2 minutes left (then the normal sign-in answers). #> param([string]$Audience, [string]$TenantId) $h = $global:PimHandoffTokens if (-not ($h -is [hashtable]) -or -not ($h.tokens -is [hashtable]) -or -not $h.tokens.Count) { return '' } if (-not "$TenantId".Trim() -or "$($h.tenantId)".Trim().ToLowerInvariant() -ne "$TenantId".Trim().ToLowerInvariant()) { return '' } $e = $h.tokens[(Get-PimHandoffAudienceKey $Audience)] if (-not $e -or -not "$($e.token)".Trim()) { return '' } if ($e.expiresUtc -le [datetime]::UtcNow.AddMinutes(2)) { return '' } return "$($e.token)" } # ---- the one entry point: get a token for a resource ---------------------- function Get-PimRestToken { [CmdletBinding()] param( [string]$Resource = 'graph', [string]$TenantId,[string]$ClientId,[string]$ClientSecret, [System.Security.Cryptography.X509Certificates.X509Certificate2]$Certificate,[string]$CertThumbprint, [switch]$UseManagedIdentity,[switch]$Interactive,[switch]$Force ) $aud = Resolve-PimRestResource -Resource $Resource # The identity is resolved BEFORE the cache is consulted, because it is part of the # cache key. See BUG-22 below -- this ordering is the fix, do not move the lookup back # above these lines. $tenant = Get-PimTenantId -TenantId $TenantId $cid = if ($ClientId) { $ClientId } elseif ($global:PIM_ClientId) { $global:PIM_ClientId } elseif ($env:AZURE_CLIENT_ID) { $env:AZURE_CLIENT_ID } else { $null } $sec = if ($ClientSecret) { $ClientSecret } elseif ($global:PIM_ClientSecret) { $global:PIM_ClientSecret } elseif ($env:AZURE_CLIENT_SECRET) { $env:AZURE_CLIENT_SECRET } else { $null } # Cert thumbprint: explicit -CertThumbprint, else the engine SPN global / env. This is # the engine's real app-only auth (SPN + certificate, no secret) -- e.g. the # PIM4EntraPS-Engine cert in LocalMachine\My / CurrentUser\My. $thumb = if ($CertThumbprint) { $CertThumbprint } elseif ($global:PIM_CertThumbprint) { $global:PIM_CertThumbprint } elseif ($env:PIM_CERT_THUMBPRINT) { $env:PIM_CERT_THUMBPRINT } else { $null } $cert = if ($Certificate) { $Certificate } else { Resolve-PimCertificate -Thumbprint $thumb } # 🔴 SEC-12 -- WHAT THE CALLER ASKED FOR, which is NOT the same as what resolution produced. # Deliberately keyed on $thumb (the thumbprint that was REQUESTED), never on $cert (the object # that was RESOLVED): a certificate that was named and could not be loaded is a FAILURE TO # HONOUR the request, not the absence of one. Reading $cert here is exactly how a missing cert # became "no explicit identity was asked for" and slid onto the (since removed) ambient command-line fallback. $explicitIdentity = [bool]("$tenant".Trim()) -and [bool]("$cid".Trim()) -and ([bool]("$sec".Trim()) -or [bool]("$thumb".Trim()) -or [bool]$Certificate) # BUG-22 (2026-08-06): this cache was keyed by the AUDIENCE ALONE -- no tenant, no client # id. In any process that touches TWO tenants, the second one silently reused the first # one's token, so every call meant for tenant B went to tenant A. That is precisely the # MSP master->slave fanout this product exists to run, and the failure is invisible: the # calls SUCCEED, against the wrong directory. # # Proven live: an S6 (local-slave) run targeting a tenant with ZERO PIM groups reported # "Groups live=85" -- the MASTER's group count -- and the verifier then found all six # groups missing from the managed tenant it was supposed to be managing. A write-shaped run would # have applied the customer's desired state to the MSP's own tenant. # # The key is now the full identity: audience + tenant + client + credential KIND (and the # cert thumbprint, so rotating a cert re-mints rather than serving the old token). No # secret material goes into the key. $mode = if ($Interactive -or $global:PIM_Interactive) { 'interactive' } elseif ($UseManagedIdentity -or $global:PIM_UseManagedIdentity -or ($env:IDENTITY_ENDPOINT -and -not $cid)) { 'mi' } elseif ($sec) { 'secret' } elseif ($cert) { "cert:$thumb" } elseif ($global:PIM_TokenProvider -is [scriptblock]) { 'signed-in' } else { 'unknown' } $key = ("$aud|$tenant|$cid|$mode").ToLowerInvariant() # same cross-scope guard as Resolve-PimRestResource: an unseeded cache must mean "cache # miss", never a thrown method call that the caller reads as "authentication failed". if (-not $script:PimTokenCache) { $script:PimTokenCache = @{} } if (-not $Force -and $script:PimTokenCache.ContainsKey($key)) { $e = $script:PimTokenCache[$key] if ($e.expiresUtc -gt (Get-Date).ToUniversalTime().AddMinutes(2)) { return $e.token } } # Framework 8.13 INSTALL-HANDOFF (REQ 100.66): the sign-in Invardia's bootstrap handed this process, already validated by # Import-PimHandoffTokens (this tenant, a USER, >5 min left) and held in memory only, answers FIRST for its audiences -- # but never for a caller that named an explicit identity, and never for an explicit browser request (-Interactive, which # may need scopes the handed-over token does not carry). An audience it does not cover falls through to the sign-in below. # 🔴 2026-10-11 (a managed-tenant install, live): NEVER for Azure SQL. The handed-over database token is minted by the bootstrap # BEFORE this run creates the SQL server and makes the signed-in admin (or its group) the server's Entra admin -- that token # carries no such membership, and the schema step failed "Login failed for user ''". The # database audience always comes from a sign-in made now (after the assignment). if (-not $explicitIdentity -and -not ($Interactive -or $global:PIM_Interactive) -and "$aud" -notmatch '(?i)database\.windows\.net') { $hk = Get-PimHandoffToken -Audience $aud -TenantId $tenant if ($hk) { return $hk } } $res = $null # Explicit interactive request (break-glass): sign in as the human up front. if ($Interactive -or $global:PIM_Interactive) { try { $res = Get-PimInteractiveToken -Audience $aud -TenantId $tenant } catch { Write-Verbose "PIM-Rest interactive auth failed for ${Resource}: $($_.Exception.Message)" } } $primaryErr = $null if (-not $res) { try { if ($UseManagedIdentity -or $global:PIM_UseManagedIdentity -or ($env:IDENTITY_ENDPOINT -and -not $cid)) { $res = Get-PimManagedIdentityToken -Audience $aud } elseif ($tenant -and $cid -and $sec) { $res = Get-PimClientSecretToken -TenantId $tenant -ClientId $cid -ClientSecret $sec -Audience $aud } elseif ($tenant -and $cid -and $cert) { $res = Get-PimClientCertToken -TenantId $tenant -ClientId $cid -Certificate $cert -Audience $aud } } catch { # 🔴 SEC-12: KEEP the reason. This was Write-Verbose only, so the one fact that explained # everything downstream was discarded at the moment it was known. $primaryErr = "$($_.Exception.Message)" Write-Verbose "PIM-Rest primary auth failed for ${Resource}: $primaryErr" } } # ===================================================================================== # 🔴 SEC-12 -- AN EXPLICIT IDENTITY THAT CANNOT BE HONOURED IS AN ERROR, NOT A CUE TO # BECOME SOMEBODY ELSE. # # MEASURED 2026-08-28: a caller asked for the internal tenant + the PIM engine # client id + its certificate thumbprint, and got back a token for ANOTHER company's tenant # (A DIFFERENT COMPANY's tenant) with a different appid. Nothing in the return value # said so. The certificate had failed to resolve, the failure went to Write-Verbose, and the # "dev convenience" command-line fallback (since removed, REQ 100.42) then minted a token for whatever # subscription happened to be that tool's DEFAULT context -- which on the dev machine is frequently another # tenant entirely. # # 🔑 THE POINT IS NOT THAT THE FALLBACK EXISTS. It is that it ran AFTER the caller had named a # tenant, a client id and a credential. That caller has said exactly who it wants to be, and # answering with a different principal is WORSE than answering with an error: the token works, # so the failure surfaces far away as "Login failed" or "permission denied" and reads like an # RBAC problem. It cost a session's time to chase, and the answer was three layers up. # # 📌 Same family as BUG-34, which fixed precisely this shape one layer down in # New-PimSqlConnection ("a valid-but-wrong-tenant token was taken, so the explicitly # configured SPN never got a turn") and left the command-line branch here standing (removed since, REQ 100.42). # ===================================================================================== if (-not $res -and $explicitIdentity) { $why = if ("$thumb".Trim() -and -not $cert) { "certificate '$thumb' was not found in CurrentUser\My or LocalMachine\My, or has no usable private key" } elseif ($primaryErr) { $primaryErr } else { 'the credential was not accepted' } throw ("PIM-Rest: REFUSING to fall back to an ambient identity for '$Resource'. " + "An explicit identity was requested (tenant '$tenant', client '$cid') and could not be honoured: $why. " + 'Fix the credential rather than letting the call authenticate as somebody else -- a token for the wrong ' + 'principal succeeds and then fails far away as a permissions error.') } # §100.42 / framework §12.17 (owner 2026-10-09: "modern single connect only"): the command-line fallback ("reuse # whoever is signed in to the CLI") is GONE. A signed-in setup run does not borrow a tool's session any more: # tools\setup\_PimSignedIn.ps1 (Set-PimSignedInGlobals) REGISTERS its token source here -- # the Invardia Support-app session's Get-InvardiaSupportToken when that session is open in the shell, otherwise the # person's own browser sign-in (Get-PimInteractiveToken, auth-code + PKCE). Only reached when NO explicit identity # was requested (the refusal above still throws for one that cannot be honoured). # 2026-10-10: a setup step's OWN child processes (e.g. the schema step) inherit PIM_HANDOFF_TOKENS from the build -- answer # from it first, so no grandchild process opens a silent browser sign-in. # 8.13: either shape is read -- the build's own (audience -> token string) or the bootstrap's (audience -> {access_token, # expires_on}) -- and a token whose tid is not the requested tenant is never returned (SEC-12). if (-not $res -and "$env:PIM_HANDOFF_TOKENS".Trim() -and -not $explicitIdentity) { try { $ho = "$env:PIM_HANDOFF_TOKENS" | ConvertFrom-Json foreach ($pp in $ho.PSObject.Properties) { if ("$($pp.Name)".TrimEnd('/').ToLowerInvariant() -eq "$aud".TrimEnd('/').ToLowerInvariant()) { # REQ 100.76: with a handed-over refresh token, Azure SQL is never answered from a pre-minted access token. if ("$aud" -match '(?i)database\.windows\.net' -and "$env:PIM_HANDOFF_REFRESH".Trim()) { continue } $hv = if ($pp.Value -is [string]) { "$($pp.Value)" } elseif ($pp.Value -and $pp.Value.PSObject.Properties['access_token']) { "$($pp.Value.access_token)" } else { '' } if (-not "$hv".Trim()) { continue } $hvTid = Get-PimTokenTenantId -Token $hv if ("$tenant".Trim() -and $hvTid -and $hvTid -ne "$tenant".Trim().ToLowerInvariant()) { Write-Verbose "PIM_HANDOFF_TOKENS: the token for $aud is for another tenant -- not used"; continue } return $hv } } } catch { Write-Verbose "PIM_HANDOFF_TOKENS unreadable: $($_.Exception.Message)" } } # REQ 100.76: a step's child process with no token source of its own mints from the handed-over refresh token -- silently, # at the moment it is asked (never a browser window from here). if (-not $res -and "$env:PIM_HANDOFF_REFRESH".Trim() -and -not $explicitIdentity -and -not ($global:PIM_TokenProvider -is [scriptblock])) { try { $res = Get-PimInteractiveToken -Audience $aud -TenantId $tenant -RefreshOnly } catch { Write-Verbose "PIM_HANDOFF_REFRESH exchange for ${Resource} failed: $($_.Exception.Message)" } } if (-not $res -and ($global:PIM_TokenProvider -is [scriptblock])) { $pt = $null try { $pt = & $global:PIM_TokenProvider $aud $tenant } catch { throw "PIM-Rest: the signed-in token source failed for '$Resource': $($_.Exception.Message)" } if ($pt -is [string]) { $pt = [pscustomobject]@{ token = $pt; expiresUtc = (Get-Date).ToUniversalTime().AddMinutes(30) } } if ($pt -and "$($pt.token)".Trim()) { # 🔒 VERIFY IT (SEC-12): asking for a tenant is not the same as being given one. $claimed = Get-PimTokenTenantId -Token "$($pt.token)" if ("$tenant".Trim() -and $claimed -and ($claimed -ne "$tenant".Trim().ToLowerInvariant())) { throw "PIM-Rest: REFUSED a signed-in token for '$Resource': it belongs to tenant $claimed, not the requested $tenant." } $res = $pt } } # LAST-RESORT interactive prompt — ONLY when an attended caller opts in via # $global:PIM_InteractiveFallback (the interactive admin tools set this; the engine / # headless cron NEVER does, so an unattended run still fails fast instead of hanging on # a browser prompt). This is what makes the admin deploy scripts "just sign me in" when # no MI/secret/cert token is available, rather than throwing. if (-not $res -and $global:PIM_InteractiveFallback) { Write-Host "PIM-Rest: no MI/secret/cert token for '$Resource' -- falling back to interactive sign-in..." -ForegroundColor Yellow try { $res = Get-PimInteractiveToken -Audience $aud -TenantId $tenant } catch { Write-Verbose "PIM-Rest interactive fallback failed for ${Resource}: $($_.Exception.Message)" } } # MI-TOKEN-TRANSIENT: carry the primary credential's own answer -- without it the record named no cause (SUP-2026-00323). if (-not $res) { throw ("PIM-Rest: could not acquire a token for '$Resource'. Provide a managed identity, ClientId + certificate (+TenantId), the Invardia Support-app session, or -Interactive (browser sign-in)." + $(if ($primaryErr) { " ($primaryErr)" } else { '' })) } $script:PimTokenCache[$key] = $res return $res.token } # ---- REST error reporting (BUG-27) ---------------------------------------- # A REST failure used to be logged as, verbatim: # PUT https://management.azure.com/.../roleEligibilityScheduleRequests/... # -> HTTP 409 : # The colon is where the reason should be. ARM had actually answered # { "error": { "code": "ReadOnlyDisabledSubscription", # "message": "The subscription '...' is disabled and therefore # marked as read only." } } # and the engine threw it away. Three wrong hypotheses were checked with live ARM # queries before replaying the same PUT by hand produced the answer instantly. # # The whole audit rests on believing what the engine reports: a scope reporting # "errors=1" with no cause is indistinguishable from a real code defect. In a # customer tenant the same line is a support round-trip instead of a fix. # --------------------------------------------------------------------------- function Get-PimRestErrorBody { <# Read the error body from a terminating REST error, from BOTH sources. 🪤 The old code was `if (ErrorDetails.Message) {...} else { read the stream }`. On Windows PowerShell 5.1 ErrorDetails.Message is frequently PRESENT BUT EMPTY, so the else-branch never ran and the stream -- which held the real answer -- was never read. Take the LONGER of the two instead of trusting either. #> [CmdletBinding()] param([Parameter(Mandatory)][object]$ErrorRecord) $fromDetails = '' try { if ($ErrorRecord.ErrorDetails) { $fromDetails = "$($ErrorRecord.ErrorDetails.Message)" } } catch {} $fromStream = '' try { $resp = $ErrorRecord.Exception.Response if ($resp) { $stream = $resp.GetResponseStream() if ($stream) { # The stream may already have been consumed; rewind when we are allowed to. try { if ($stream.CanSeek) { $stream.Position = 0 } } catch {} $sr = New-Object System.IO.StreamReader($stream) try { $fromStream = $sr.ReadToEnd() } finally { $sr.Dispose() } } } } catch {} if ("$fromStream".Trim().Length -gt "$fromDetails".Trim().Length) { return $fromStream } return $fromDetails } function Get-PimRestErrorDetail { <# PURE. Turn an ARM/Graph error body into the SHORT reason a human needs. Returns " -- ", or $null when the body carries nothing usable (the caller must then say so explicitly rather than printing an empty reason). 🔒 CODE + MESSAGE ONLY, NEVER THE WHOLE BODY. The raw body can carry subscription ids, principal ids and request ids; it is also mostly noise. code+message is the part that ends the diagnosis, so it is the right amount to log. Handles the shapes both APIs actually return: ARM { "error": { "code": ..., "message": ... } } Graph { "error": { "code": ..., "message": ..., "innerError": {...} } } flat { "code": ..., "message": ... } odata { "odata.error": { "code": ..., "message": { "value": ... } } } non-JSON (HTML error pages, plain text) -- returned trimmed + truncated. #> [CmdletBinding()] param([Parameter()][AllowNull()][AllowEmptyString()][string]$Body, [int]$MaxLength = 400) $raw = "$Body".Trim() if (-not $raw) { return $null } $obj = $null try { $obj = $raw | ConvertFrom-Json } catch { $obj = $null } # Property lookup is CASE-INSENSITIVE. `-contains` is not, and Azure is not consistent: # ARM/Graph send "message", some services send "Message". A case-sensitive check would # silently miss those and fall through to printing the raw body. $prop = { param($o, $name) try { foreach ($p in $o.PSObject.Properties) { if ("$($p.Name)" -ieq "$name") { return $p.Value } } } catch {} return $null } if ($obj) { $err = $null foreach ($k in @('error', 'odata.error')) { $v = & $prop $obj $k if ($null -ne $v) { $err = $v; break } } if ($null -eq $err) { $err = $obj } # flat { code, message } $code = "$(& $prop $err 'code')".Trim() $m = & $prop $err 'message' $msg = '' if ($null -ne $m) { # odata nests the text one level deeper: message.value if ($m -isnot [string]) { $inner = & $prop $m 'value' $msg = if ($null -ne $inner) { "$inner".Trim() } else { "$m".Trim() } } else { $msg = "$m".Trim() } } if ($code -or $msg) { $joined = if ($code -and $msg) { "$code -- $msg" } elseif ($code) { $code } else { $msg } $joined = ($joined -replace '\s+', ' ').Trim() if ($joined.Length -gt $MaxLength) { $joined = $joined.Substring(0, $MaxLength) + '...' } return $joined } # Valid JSON that carries no code and no message -- e.g. "{}". Echoing it back would # print "HTTP 409 : {}", which is exactly as useless as the empty colon this whole # function exists to eliminate. Report NOTHING USABLE and let the caller say so. $hasAny = $false try { $hasAny = @($obj.PSObject.Properties).Count -gt 0 } catch {} if (-not $hasAny) { return $null } } # Not JSON (HTML error page, plain text), or JSON whose useful part we could not name: # return it trimmed + truncated. Better than nothing, and still bounded -- an error # page must not flood the log. $flat = ($raw -replace '\s+', ' ').Trim() if (-not $flat) { return $null } if ($flat.Length -gt $MaxLength) { $flat = $flat.Substring(0, $MaxLength) + '...' } return $flat } # ---- data plane ----------------------------------------------------------- # Every REST call is bounded (see Invoke-PimRest). $global:PIM_RestTimeoutSec overrides; floor 10 s, default 180 s. function Get-PimRestTimeoutSec { $v = 0; if ($global:PIM_RestTimeoutSec) { [void][int]::TryParse("$($global:PIM_RestTimeoutSec)", [ref]$v) } if ($v -ge 10) { return $v } return 180 } function Test-PimRestTimeoutError { # pwsh 7: HttpClient's timeout surfaces as TaskCanceledException / TimeoutException ("The request was canceled due to the # configured HttpClient.Timeout"); PS 5.1: WebException status Timeout ("The operation has timed out"). param($ErrorRecord) $e = $null; try { $e = $ErrorRecord.Exception } catch { } while ($e) { if ($e -is [System.TimeoutException] -or $e -is [System.Threading.Tasks.TaskCanceledException]) { return $true } if ($e -is [System.Net.WebException] -and $e.Status -eq [System.Net.WebExceptionStatus]::Timeout) { return $true } $e = $e.InnerException } return ("$($ErrorRecord)" -match 'HttpClient\.Timeout|operation has timed out|request was canceled') } function Test-PimRestNetworkError { # UPDATER-ARM-RETRY (2026-10-21, internal): a call that failed BELOW HTTP (no status code) on a one-off network fault -- # "Cannot assign requested address" (no free local port), connection refused / reset, a name that did not resolve. It failed # the updater's PATCH of ca-pim-manager once and nothing retried. Connect-level faults never reached the service, so any method # is safe to send again; a reset AFTER the request was sent is retried only for an idempotent method (never a POST). param($ErrorRecord, [string]$Method = 'GET') $connect = 'Cannot assign requested address|No connection could be made|actively refused|Connection refused|No such host is known|Name or service not known|nodename nor servname|Temporary failure in name resolution|Network is unreachable|NameResolutionFailure|ConnectFailure' $reset = 'forcibly closed|Connection reset|connection was closed|transport connection|Broken pipe|An error occurred while sending the request|ReceiveFailure|SendFailure' $text = New-Object System.Collections.Generic.List[string] $e = $null; try { $e = $ErrorRecord.Exception } catch { } $isSock = $false while ($e) { $text.Add("$($e.Message)") if ($e -is [System.Net.Sockets.SocketException]) { $isSock = $true } if ($e -is [System.Net.WebException]) { $text.Add("$($e.Status)") } $e = $e.InnerException } $text.Add("$ErrorRecord") $all = $text -join ' | ' if ($all -match $connect) { return $true } $idem = @('GET','HEAD','PUT','PATCH','DELETE','OPTIONS') -contains "$Method".ToUpperInvariant() if ($idem -and ($isSock -or $all -match $reset)) { return $true } return $false } function Get-PimRestNetworkRetryMax { <# PURE. Extra tries for a network fault (a literal). #> return 3 } # §33.0 MORE TELEMETRY round 2 (owner "add relevant. i want data"): RETRIES PER HTTP CODE. Every 429 / 5xx retry of a GRAPH # call (Invoke-PimRest's backoff loop and the $batch sub-requests) is counted in this process ($global: -- a dot-sourced # function's $script: follows the caller's scope); the scheduler persists the counts after every job run into hourly # buckets (pim.Settings 'GraphRetryCounts', Save-PimRestRetryCounts), and the heartbeat sends graphThrottled24h = the 429s # of the last 24 hours. Counts only -- no URL, no tenant, no body. function Add-PimRestRetryCount { param([int]$Code, [string]$Resource = 'graph') try { if ("$Resource" -ne 'graph') { return } $k = if ($Code -eq 429) { 'r429' } elseif ($Code -ge 500) { 'r5xx' } else { '' } if (-not $k) { return } if (-not ($global:PimRestRetryCounts -is [hashtable])) { $global:PimRestRetryCounts = @{ r429 = [long]0; r5xx = [long]0 } } $global:PimRestRetryCounts[$k] = [long]$global:PimRestRetryCounts[$k] + 1 } catch { } } function Get-PimRestRetryCounts { # A COPY of this process's not-yet-persisted counts: @{ r429; r5xx } (zeros when nothing was counted). $c = $global:PimRestRetryCounts if (-not ($c -is [hashtable])) { return @{ r429 = [long]0; r5xx = [long]0 } } return @{ r429 = [long]$c['r429']; r5xx = [long]$c['r5xx'] } } function Get-PimRestRetryBucketHours { <# PURE. How many hourly buckets the record keeps (a literal). #> return 48 } function Merge-PimRestRetryBuckets { <# PURE. The stored record (raw JSON text, or $null / '' for none) plus -Add @{ r429; r5xx } in the bucket of -NowUtc's hour -> the new record as JSON text: { sinceUtc; hours = { 'yyyy-MM-ddTHH:00:00Z' = { r429; r5xx } } }. Buckets older than Get-PimRestRetryBucketHours are dropped; sinceUtc (when counting started here) is kept. A stored value that is not the record's shape starts a new record. Built as text (no DateTime round-trip through ConvertFrom-Json on PS 7). #> param([AllowNull()][string]$Existing, [hashtable]$Add = @{}, [datetime]$NowUtc = [datetime]::UtcNow) $now = $NowUtc.ToUniversalTime() $cut = $now.AddHours(-1 * (Get-PimRestRetryBucketHours)) $inv = [System.Globalization.CultureInfo]::InvariantCulture $styles = [System.Globalization.DateTimeStyles]::AssumeUniversal -bor [System.Globalization.DateTimeStyles]::AdjustToUniversal $since = $now.ToString('yyyy-MM-ddTHH:mm:ssZ', $inv) $hours = New-Object 'System.Collections.Generic.SortedDictionary[string,object]' ([System.StringComparer]::Ordinal) if ("$Existing".Trim()) { try { $o = "$Existing" | ConvertFrom-Json if ($null -ne $o -and $o.PSObject.Properties['hours']) { $sv = $o.sinceUtc if ($sv -is [datetime]) { $since = $sv.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ', $inv) } elseif ("$sv".Trim()) { $d0 = [datetime]::MinValue; if ([datetime]::TryParse("$sv", [System.Globalization.CultureInfo]::InvariantCulture, $styles, [ref]$d0)) { $since = $d0.ToString('yyyy-MM-ddTHH:mm:ssZ', $inv) } } if ($null -ne $o.hours) { foreach ($p in @($o.hours.PSObject.Properties)) { $d = [datetime]::MinValue if (-not [datetime]::TryParse("$($p.Name)", [System.Globalization.CultureInfo]::InvariantCulture, $styles, [ref]$d)) { continue } if ($d -lt $cut) { continue } $a = 0L; $b = 0L [void][long]::TryParse("$($p.Value.r429)", [ref]$a); [void][long]::TryParse("$($p.Value.r5xx)", [ref]$b) $hours[$d.ToString('yyyy-MM-ddTHH:00:00Z', $inv)] = @{ r429 = [math]::Max(0L, $a); r5xx = [math]::Max(0L, $b) } } } } } catch { $hours.Clear() } } $key = $now.ToString('yyyy-MM-ddTHH:00:00Z', $inv) $ad = 0L; $bd = 0L; [void][long]::TryParse("$($Add['r429'])", [ref]$ad); [void][long]::TryParse("$($Add['r5xx'])", [ref]$bd) if ($ad -gt 0 -or $bd -gt 0) { $cur = if ($hours.ContainsKey($key)) { $hours[$key] } else { @{ r429 = 0L; r5xx = 0L } } $hours[$key] = @{ r429 = [long]$cur.r429 + [math]::Max(0L, $ad); r5xx = [long]$cur.r5xx + [math]::Max(0L, $bd) } } $parts = @(foreach ($k in $hours.Keys) { '"{0}":{{"r429":{1},"r5xx":{2}}}' -f $k, [long]$hours[$k].r429, [long]$hours[$k].r5xx }) return ('{{"sinceUtc":"{0}","hours":{{{1}}}}}' -f $since, ($parts -join ',')) } function Save-PimRestRetryCounts { <# Persist this process's counts into pim.Settings 'GraphRetryCounts' (compare-and-set, 5 tries -- several tick executions run at once) and zero what was saved. Writes ONLY when something was counted (a quiet run touches nothing -- no read, no write), so "no record" = no retry was ever recorded here. Never throws. Returns @{ saved; reason }. -Read { param($cs,$name) } / -Write { param($cs,$name,$new,$expected) -> rows }: test seams. #> param([string]$ConnectionString = '', [datetime]$NowUtc = [datetime]::UtcNow, [scriptblock]$Read = $null, [scriptblock]$Write = $null) try { $c = Get-PimRestRetryCounts $any = ([long]$c.r429 + [long]$c.r5xx) -gt 0 if (-not $any) { return @{ saved = $false; reason = 'nothing counted' } } $cs = "$ConnectionString" if (-not $cs.Trim() -and (Get-Command Get-PimSqlSettingsConnectionString -ErrorAction SilentlyContinue)) { $cs = "$(Get-PimSqlSettingsConnectionString)" } if (-not $cs.Trim()) { return @{ saved = $false; reason = 'no SQL store' } } if (-not $Read) { if (-not (Get-Command Get-PimSqlSettingRaw -ErrorAction SilentlyContinue)) { return @{ saved = $false; reason = 'the settings reader is not loaded' } } $Read = { param($s, $n) Get-PimSqlSettingRaw -ConnectionString $s -Name $n } } if (-not $Write) { if (-not (Get-Command Set-PimSqlSettingIfUnchanged -ErrorAction SilentlyContinue)) { return @{ saved = $false; reason = 'the settings writer is not loaded' } } $Write = { param($s, $n, $new, $exp) Set-PimSqlSettingIfUnchanged -ConnectionString $s -Name $n -NewValueJson $new -ExpectedValueJson $exp } } for ($i = 0; $i -lt 5; $i++) { $raw = & $Read $cs 'GraphRetryCounts' $new = Merge-PimRestRetryBuckets -Existing $raw -Add $c -NowUtc $NowUtc $n = 0; try { $n = [int](@(& $Write $cs 'GraphRetryCounts' $new $raw) | Select-Object -Last 1) } catch { $n = 0 } if ($n -ge 1) { if ($global:PimRestRetryCounts -is [hashtable]) { $global:PimRestRetryCounts['r429'] = [math]::Max(0L, [long]$global:PimRestRetryCounts['r429'] - [long]$c.r429) $global:PimRestRetryCounts['r5xx'] = [math]::Max(0L, [long]$global:PimRestRetryCounts['r5xx'] - [long]$c.r5xx) } return @{ saved = $true; reason = '' } } } return @{ saved = $false; reason = 'another writer kept winning -- kept for the next run' } } catch { return @{ saved = $false; reason = "$($_.Exception.Message)" } } } function Invoke-PimRest { [CmdletBinding()] param( [string]$Method = 'GET', [Parameter(Mandatory)][string]$Url, [object]$Body, [string]$Resource = 'graph', [hashtable]$Headers = @{}, [switch]$All, # follow @odata.nextLink / nextLink, aggregate .value [int]$MaxRetry = 5, # Authenticate THIS call as the host's managed identity even when an engine SPN is configured # (mail send, operator decision 2026-09-13: the hosted engine sends as its managed identity). [switch]$UseManagedIdentity ) # Keep a scheduler lease alive through a long live read (minutes of paged calls). No-op outside # a leased tick; time-gated in Invoke-PimSchedulerLeaseHeartbeat (PIM-Scheduler.ps1). if ($global:PIM_LeaseHeartbeat -and (Get-Command Invoke-PimSchedulerLeaseHeartbeat -ErrorAction SilentlyContinue)) { [void](Invoke-PimSchedulerLeaseHeartbeat) } $token = if ($UseManagedIdentity) { Get-PimRestToken -Resource $Resource -UseManagedIdentity } else { Get-PimRestToken -Resource $Resource } $h = @{ Authorization = "Bearer $token" } + $Headers $agg = New-Object System.Collections.Generic.List[object] $next = $Url while ($next) { $attempt = 0; $netTry = 0 while ($true) { try { # 🔴 2026-10-02: NO -TimeoutSec = wait FOREVER on pwsh 7. One Graph call that never answered froze the hybrid worker's # continuous AD sync for 22 hours (process alive, task "Running", IgnoreNew -> never restarted). Every call is bounded now. $args = @{ Method = $Method; Uri = $next; Headers = $h; TimeoutSec = (Get-PimRestTimeoutSec) } # (see Get-PimRestErrorBody / Get-PimRestErrorDetail above for the failure path) if ($null -ne $Body -and $Method -ne 'GET') { $args.Body = if ($Body -is [string]) { $Body } else { $Body | ConvertTo-Json -Depth 20 } $args.ContentType = 'application/json' } $resp = Invoke-RestMethod @args break } catch { $code = $null; try { $code = [int]$_.Exception.Response.StatusCode } catch {} # BUG-27: surface the API error body. Try BOTH sources and keep the longer of the # two -- on 5.1 ErrorDetails.Message is frequently present but EMPTY, and the old # `if/else` took it and never looked at the stream, which is how a real ARM # explanation became the bare log line "-> HTTP 409 :" (note the empty colon). $body = Get-PimRestErrorBody -ErrorRecord $_ # retry transient + freshly-created-principal replication (ARM 400 PrincipalNotFound) $isReplDelay = ($code -eq 400 -and "$body" -match 'PrincipalNotFound|does not exist in the directory') # a call that hit -TimeoutSec is retried ONCE (a second full wait is the most a stuck endpoint may cost), then fails $isTimeout = (-not $code) -and (Test-PimRestTimeoutError -ErrorRecord $_) if ($isTimeout -and $attempt -lt 1) { $attempt++; continue } if ($isTimeout) { throw "$Method $next -> timed out after $(Get-PimRestTimeoutSec) s (twice) -- the service did not answer" } # UPDATER-ARM-RETRY: a one-off network fault (no HTTP status) is tried again with a short backoff (5, 10, 20 s) if ((-not $code) -and (Test-PimRestNetworkError -ErrorRecord $_ -Method $Method)) { if ($netTry -lt (Get-PimRestNetworkRetryMax)) { Start-Sleep -Seconds ([int](5 * [Math]::Pow(2, $netTry))); $netTry++; continue } } if (($code -eq 429 -or $code -ge 500 -or $isReplDelay) -and $attempt -lt $MaxRetry) { if ($code -eq 429 -or $code -ge 500) { Add-PimRestRetryCount -Code ([int]$code) -Resource $Resource } # §33.0 round 2 $wait = [Math]::Min(60, [Math]::Pow(2, $attempt + 1)) try { $ra = [int]("$($_.Exception.Response.Headers['Retry-After'])"); if ($ra -gt 0) { $wait = $ra } } catch {} Start-Sleep -Seconds $wait; $attempt++; continue } # section 9 missing-role hint: on a 403 / insufficient-privileges, append the EXACT # Graph app-role to grant (engine SPN, app-only) so the operator isn't left # guessing. Best-effort: only when the hint helper is loaded (PIM-Functions / # PIM-AuthDiagnostics dot-sourced) and the failure is a permissions failure. $hintText = '' if (Get-Command Get-PimMissingRoleHint -ErrorAction SilentlyContinue) { try { $appOnly = -not [bool]$global:PIM_Interactive $h = Get-PimMissingRoleHint -Path $next -StatusCode ([int]("$code")) -ErrorBody "$body" -AppOnly:$appOnly if ($h) { $hintText = " >> $($h.Hint)" } } catch {} } # BUG-27: report error.code + error.message, NOT the raw body. The raw body can # carry ids and is usually mostly noise; code+message is the part that ends the # diagnosis. And a body that yields nothing usable must SAY SO -- "HTTP 409 :" # with an empty reason reads as "no reason exists", which sent a whole session # chasing three wrong hypotheses while ARM had the answer all along # ("ReadOnlyDisabledSubscription -- the subscription is disabled and therefore # marked as read only"). $detail = Get-PimRestErrorDetail -Body $body # 2026-09-21 (operator: "can you provide graph or arm errors in here as i have nothing to work with"): the # thrown message stays code+message (above), but the full body of the LAST failure is kept for a caller that # needs more -- e.g. a policy-rule PATCH that adds the request-id and what it sent (Invoke-PimPolicyRulePatch). # 100.50: the Retry-After of that failure too (seconds; 0 = none) -- an ARM write refused with 409 OperationInProgress # is waited out by Invoke-PimArmBusyRetry (PIM-ArmContainerApps.ps1), which honours it when the service sent one. $raLast = 0 try { $d = $_.Exception.Response.Headers.RetryAfter.Delta; if ($d) { $raLast = [int][Math]::Ceiling($d.TotalSeconds) } } catch { } if (-not $raLast) { try { [void][int]::TryParse("$($_.Exception.Response.Headers['Retry-After'])", [ref]$raLast) } catch { } } $global:PimLastRestError = [pscustomobject]@{ method = $Method; url = $next; status = $code; body = "$body"; atUtc = [datetime]::UtcNow; retryAfter = $raLast } if ($detail) { throw "$Method $next -> HTTP $code : $detail$hintText" } throw "$Method $next -> HTTP $code : (no error body returned by the service)$hintText" } } if (-not $All) { return $resp } if ($null -ne $resp.value) { foreach ($v in $resp.value) { $agg.Add($v) } } else { $agg.Add($resp) } $next = $null if ($resp.'@odata.nextLink') { $next = $resp.'@odata.nextLink' } elseif ($resp.nextLink) { $next = $resp.nextLink } } return $agg.ToArray() } function ConvertTo-PimSdkShape { # Make a Graph REST object look like a Graph PowerShell SDK object: add a # PascalCase alias for every camelCase property (userPrincipalName -> # UserPrincipalName, displayName -> DisplayName, id -> Id) so existing engine # filters/consumers that expect SDK casing keep working over pure REST. param([Parameter(ValueFromPipeline)][object]$InputObject) process { if ($null -eq $InputObject) { return } $o = [ordered]@{} foreach ($p in $InputObject.PSObject.Properties) { $o[$p.Name] = $p.Value if ($p.Name.Length -ge 1) { $pascal = $p.Name.Substring(0,1).ToUpperInvariant() + $p.Name.Substring(1) if (-not $o.Contains($pascal)) { $o[$pascal] = $p.Value } } } [pscustomobject]$o } } function Invoke-PimGraph { param([string]$Method='GET',[Parameter(Mandatory)][string]$Path,[object]$Body,[switch]$All,[switch]$Beta,[hashtable]$Headers=@{},[switch]$UseManagedIdentity) $base = if ($Beta) { 'https://graph.microsoft.com/beta' } else { 'https://graph.microsoft.com/v1.0' } $url = if ($Path -match '^https?://') { $Path } else { "$base$Path" } Invoke-PimRest -Method $Method -Url $url -Body $Body -Resource 'graph' -All:$All -Headers $Headers -UseManagedIdentity:$UseManagedIdentity } function Test-PimGraphBatchReadsEnabled { # Kill switch for the batched live reads (pim.Settings / $global:PIM_ / env 'GraphBatchReads'). # ON by default; 'false' makes every caller take its original one-request-per-path loop. $v = $null if (Get-Command Get-PimPolicySetting -ErrorAction SilentlyContinue) { try { $v = Get-PimPolicySetting -Name 'GraphBatchReads' -Default $null } catch { $v = $null } } elseif ($null -ne $global:PIM_GraphBatchReads) { $v = $global:PIM_GraphBatchReads } elseif ("$env:PIM_GraphBatchReads".Trim()) { $v = $env:PIM_GraphBatchReads } return -not ("$v".Trim() -match '^(?i)(false|0|no|off|disabled?)$') } function Invoke-PimGraphBatchGet { <# Run many Graph GETs through /$batch (20 per round-trip, Graph's maximum) and return ONE result per input path, IN INPUT ORDER: { path; ok; status; items; body; error }. items = what `Invoke-PimGraph -All -Path ` returns: the aggregated .value across every @odata.nextLink page, or the body itself when it carries no .value. body = what `Invoke-PimGraph -Path ` returns (the first page / the single object). A sub-request that is throttled (429), failed server-side (5xx) or came back missing is collected; the largest Retry-After is waited ONCE (max 30 s) and the collected items are sent again as a batch, up to 3 rounds (§70.9). Only what is still unanswered after that is re-run on its own through Invoke-PimGraph, which owns the retry/backoff -- so a batch never loses a read that the sequential loop would have got. $script:PimBatchStats holds the counters of the last call. Any other 4xx is a final failure (ok=$false, error set), exactly what the sequential loop's `try { Invoke-PimGraph ... } catch { }` produced. A $batch response without `responses` (a proxy, a stub, a host that does not batch) falls back to the sequential path for that slice. #> param([Parameter(Mandatory)][AllowEmptyCollection()][string[]]$Paths, [ValidateRange(1, 20)][int]$BatchSize = 20, # SCHEMA-QUIESCE-LONG-SNAPSHOT (2): a long read (the snapshot's ~36k group reads on a big tenant) says it is alive. # With a label, one "[graph-batch]