Arcade 1.10.0

This release makes operator RBAC mandatory, adds an optional tool-discovery stack, and carries a large application delta: two new toolkits, a tool-execution history surface, a fourth project role, and outbound-address gating for LLM model endpoints.

Read Upgrade notes before upgrading. An existing deployment needs a one-time step on chart 1.9.8 first, and there is no way to perform it after upgrading.

This release applies schema changes. Migrations run automatically on helm upgrade unless <service>.database.migrations is false.

• Engine — adds an index so the scheduler's poll seeks directly to the next due execution instead of scanning a table that grows with every run ever recorded. Required: the engine refuses to start against the older schema. A second index is created and then dropped again within this same release; the net effect is one new index, but the upgrade still pays to build the discarded one first, because migrations apply in order.
• Coordinator and experience-api — none.

Both indexes build concurrently, so no exclusive lock is taken and tool execution keeps serving throughout. Expect a transient rise in database CPU while they build; on a large tool_execution table this is measurable.

Interrupting the build needs manual recovery. A concurrent index build that is cut short — by the migration job's deadline, a node drain, or an eviction — leaves an invalid index behind and marks the schema version dirty. Retries do not clear it: the job fails on the dirty version, and forcing past that fails again because the invalid index still holds the name. Because this runs as a pre-upgrade hook, the release aborts before any workload is touched, and every later upgrade stops at the same point until someone drops the invalid index and resets the version by hand.

Operators with a large tool_execution table should raise jobs.activeDeadlineSeconds or build the index out of band before upgrading. Small tables finish in seconds and need nothing.

Rolling the chart back does not run down-migrations, so the new index simply stays; the previous engine image runs against it without trouble.

• LLM model endpoints resolving to private, loopback, link-local, or metadata addresses are refused, and redirects are never followed.
• Registering a custom model through the admin API requires an https base URL resolving to a public address.
• Refresh tokens presented at the OAuth2 token endpoint were being written into exported trace data. Where trace export is enabled, treat refresh tokens issued before this release as disclosed; they have a one-year lifetime and no revocation store.
• Session cookies, Authorization headers, and x-api-key values are no longer copied onto exported traces. Spans already exported keep the unredacted values for their retention window.
• Reading an organization's subscription, usage, upcoming schedule, upcoming invoice, or recent invoices now requires the organization-admin billing-read permission. Previously any member could fetch them, including the hosted invoice URL and PDF link.
• Google Docs content can no longer forge structure in what the model reads: text runs, front-matter values, comments, headings, and line-leading markers are escaped, so prose written by whoever owns a shared document is read as content rather than as markup.
• A pasted Google Sheets link whose query string or fragment contained a document-id segment resolved to the wrong spreadsheet — reachable through the write tools, so an edit could land in a workbook the user never named.

• ServiceNow toolkit — 17 tools covering incident creation, record search and update, comments, CMDB, knowledge, catalog browsing and ordering, and user and assignment-group lookup; four write to the instance. Inert until a servicenow OAuth2 provider is configured and the SERVICENOW_INSTANCE_URL secret is set.
• Mixpanel toolkit — 15 read-only analytics tools (trends, retention, funnels, cohorts, profiles, saved reports). All four of the service-account username, service-account secret, project id, and region secrets must be set, including for US projects, or no tool will execute.
• Optional tool-discovery stack behind discovery.enabled, default off. A default upgrade renders no new workloads and leaves the platform Redis and Postgres untouched. Known limitation: the discovery chart names every resource after the Helm release, so installing it as a subchart collides with the platform's Redis, Postgres, and app-secret names — deploy it as its own release in the same namespace until a name-prefix option exists.
• Recorded tool-execution payloads can be encrypted at rest behind engine.encryptExecutionPayloads, default off. Recorded inputs and outputs are plain JSON today, so anyone holding database credentials reads end-user data in the clear. The key lives under encryption.executionPayloadKeyExistingSecret, separate from the root key so holding it does not also decrypt OAuth tokens and stored secrets, and is generated and preserved like the root key when unset. Reads always decrypt, so the flag can be turned on and off without stranding rows.
• Tool Executions page — per-project list of recorded tool runs, filterable by tool, toolkit, user, outcome, failure reason, and time range. Recorded inputs and outputs stay withheld behind per-run toggles requiring project-admin authority. Empty unless execution recording is enabled.
• Organization Logging Policy — turn tool-execution recording on or off across every project and set the retention window, behind a new organization-admin permission.
• Project Viewer, a fourth built-in project role, assignable in role pickers and the invite flow and settable as an SSO provider's just-in-time default. The name understates it: it holds every project read plus tool execution, and lacks only the developer writes. A new in-app sheet shows the full role-by-capability matrix.
• Opening a tool run's recorded inputs or outputs writes an audit entry naming the reader and which halves were disclosed. Audit consumers will see a new viewed action and a tool_execution resource type; the entry carries no payload.
• rbac.openfga.database.host accepts a comma-separated multi-host Postgres endpoint for failover instead of failing the render, and targets the writer rather than a read replica. Bracketed IPv6 literals are rejected inside a multi-host list; single-host IPv6 still works.
• Deployment attempts emit completed and failed counters, the failure counter labelled with the stage that broke, so a deploy path broken for everyone can be alerted on rather than discovered by hand.
• Zoom gains meeting search across scheduled and cloud-recorded meetings, recording access, transcripts, AI meeting summaries, and a connection check.
• Microsoft Teams gains 13 tools: create, cancel, get and list meetings; list and read meeting transcripts; edit, delete and react to chat and channel messages; and chat membership management.
• Linear gains document tools (list, search, read, create, update, trash), attachment upsert and removal across issues, projects and initiatives, and a tool that reads the bytes behind Linear-hosted asset URLs.
• Outlook Calendar gains a free/busy availability tool for up to 20 people over a window shorter than 62 days, resolved against the caller's mailbox timezone. No new consent needed.
• PagerDuty incidents can be acknowledged, resolved, reopened and reassigned.

• Coordinator authorization is always decided by the policy decision point. The per-organization opt-out is gone, along with the CLI commands that enabled it and repaired role bindings. See Upgrade notes.
• History listings that mislabelled immediate runs as scheduled with a retry status now report them correctly.
• The scheduled-tools list and detail endpoints answer 410 Gone.
• A caller that disconnects mid-request — most often one abandoning the authorization-status long poll — resolves to 499 with a debug-level log instead of 500. Engine 5xx rates drop noticeably; alert thresholds tuned to the old baseline will be loose after upgrade. Malformed gateway IDs (now 404) and plugin hook priority conflicts (now 409) leave the 5xx population too.
• A revoked or deleted dashboard session stops authorizing within about 5 seconds instead of up to 5 minutes, at the cost of a session-table read on nearly every authenticated dashboard request. There is no chart value to tune this.
• The built-in list-apps tool and the connection initialization list return one entry per toolkit rather than one per auth provider: toolkits needing no authorization appear for the first time, and toolkits sharing a provider are listed separately instead of collapsing.
• Clients on the sessionless connection transport see and can call the same meta-tools as session clients where tool recommendation is enabled.
• The static gateway bound to engine config no longer appears in any project's gateway list in the dashboard. The API still returns it in every scope.
• Creating a Linear issue without a state on a team with the triage inbox enabled is refused with a message naming the destination and the selectable states, rather than filing it into Triage where it is hidden from board and project views.
• Linear project and initiative reads return external links and documents as separate fields with truncation flags instead of one combined list; project issue-attachments are no longer surfaced.
• Google Docs edits that name no tab on a multi-tab document are refused rather than applied to whichever tab the API picks; the edit response field names changed, searches report when a page came back partial, and rate limits are reported as retryable instead of as permission failures.
• Google Sheets tools accept a pasted spreadsheet URL everywhere, refuse a URL naming a specific tab or range while handing back the file id, reject Docs/Slides/Forms/folder links with a reason, and report throttling as retryable.
• Zoom's upcoming-meetings tool returns a typed, date-windowed, paginated result instead of the raw provider response, and an out-of-range limit is refused with a message naming the accepted range rather than silently clamped.
• The built-in manage-authorization tool is governed only by its engine config switch — with it on, every gateway that can reach a tool needing end-user authorization lists it, with no per-organization opt-in. Still defaults off.
• Audit Logs and Tool Executions sit under an Observability group in the project sidebar, and the organization Execution Logging page is renamed Logging Policy. URLs are unchanged.
• Walmart product search moves from page to limit/offset with a next_offset echo, at a new major toolkit version. Callers passing page or reading the old page metadata must migrate.
• The playground tool selector and chat toolkit picker are replaced by a single server picker dialog; tool lookup matches case-insensitively and with or without a version suffix.

• Managed worker deployments failed at admission on clusters applying a Localhost seccomp profile by default — no pod and no revision were created and the deploy timed out after five minutes. Tenant pods no longer force a seccomp profile and inherit whatever the cluster applies.
• Recorded execution timings were 1000× too large — a 652 ms call showed as roughly 11 minutes — and immediate runs carried a bogus scheduled-for timestamp. Rows written before the upgrade keep the inflated values and cannot be corrected.
• Immediate tool runs that failed were recorded as completed, so filtering history by outcome returned failures as successes and returned nothing for failure. Runs that stopped for end-user authorization are now recorded as failures.
• A failed identity-provider key refresh no longer wipes the cached signing keys — one transient fetch failure previously rejected every bearer token with 401 until the next hourly refresh. Routine credential rejections drop from error to warning, sharply reducing error-reporting volume from that source.
• An authorization request whose provider rejected the code exchange stayed pending forever, so the status poll never resolved. It is now marked failed immediately and the next authorization start creates a fresh flow.
• Editing a native plugin's configuration only took effect on the replica that handled the write, leaving other replicas enforcing the previous rules until restart. Only observable with two or more engine replicas.
• One unusable server no longer hides a project's entire server list. A remote MCP server whose stored header referenced a secret it no longer carried failed the whole listing, and the same failure blocked opening that server to repair it — locking the project out of Servers with no route back but a raw API call. Requirements are now computed per server, so a broken one reports itself and the rest still load.
• An org-wide GitHub pull-request listing now reports which repository each result belongs to. It previously returned none, and pull-request numbers collide across repositories, so a follow-up read, comment, or merge could land on the wrong pull request.
• A first install or upgrade with unpinned OpenFGA store and model ids no longer aborts the release with the seed job failing; the hook resolves those ids before it runs, and its deadline now covers that wait.
• Deleting a project removes its authorization records instead of leaving them behind permanently, and a policy-decision-point outage during deletion no longer turns a completed delete into a 500. Orphans already accumulated are not swept.
• A stored secret whose key casing differs from the casing a tool declares now satisfies that requirement — such secrets were reported missing in the catalog even though execution found them, leaving a tool that ran but never showed ready.
• Managed worker pods report their environment, so their logs and traces reach the collector instead of being discarded.
• A managed deployment could report itself running before its platform application id was stored, so a client that polled status and immediately fetched logs, restarted, or paused got a 500 for a few milliseconds.
• The Connected Apps page no longer crashes to a full-page error for projects with a custom OAuth 2.0 provider — those render with generic branding. High relevance for self-hosted installs, where custom providers are the norm.
• Opening a project a user has no access to degrades to per-widget unavailable placeholders instead of replacing the whole page with an error.
• Saving an MCP server or an OAuth provider with an invalid value inside a collapsed Advanced section no longer fails silently — the section opens, the rejected field takes focus, and the server's own message is shown on it.
• Dashboard page loads no longer stall behind a queue of duplicate identity-token refreshes; concurrent requests for the same user on the same pod share one refresh. One observed burst serialized six refreshes at 6.6s through 14.6s.
• Bumping a toolkit's version no longer silently empties a saved chat's tool selection on the next load.
• The connected-users provider filter labels each option with its provider id instead of listing several identical "OAuth 2.0" rows, and switching organization or project resets the table's filters, sorting, and pagination.
• A newly signed-up user who was invited to a project now sees that invitation on the onboarding screen.
• Where billing is enabled, users below organization admin no longer see a broken usage card on the project home page.
• Creating or updating a plugin whose hook claims a priority already held on that hook point returns 409 with a generic message instead of an untyped 500 that echoed the conflicting hook's name back to the caller.
• Microsoft Teams: reading a team whose description is unset no longer fails with a validation error, an empty message id no longer returns a fabricated success payload, a class of channel ids is no longer rejected as chat ids, and forwarded-message replies no longer fail upstream.
• Google Slides template fills no longer report a clean result when a placeholder-carrying master was dropped for size, oversized deck reads drop whole slides and name what was left out instead of truncating silently, an expired token now says to reauthorize, and a failed create no longer reports both "no changes applied" and "applied successfully" in one message.
• A Linear issue or cycle with documents attached reports them; previously the field was absent entirely, indistinguishable from having none, so an agent would answer "no documents" with confidence.

• The rbac.enabled value. Operator RBAC is always deployed; see Upgrade notes.
• The scheduled-tools list and detail endpoints (now 410 Gone).
• The coordinator CLI's operator-RBAC enable, repair, and system-flag commands. Only rekey remains.
• Linear's GitHub-link tool — attach GitHub URLs through the attachment upsert tool, which auto-detects them.
• The always-true access-check field on the Google Docs and Google Slides identity tools.

• rbac.enabled is removed and operator RBAC always deploys; rbac.openfga.database.migrations now defaults on. Existing deployments must backfill grants on 1.9.8 first.
• engine.llmSsrfAllowlist is new and ships ['*']. Narrow it.
• A pinned rbac.openfga.modelId must be cleared.
• rbac.openfga.database.host accepts a multi-host endpoint.
• discovery.enabled is new and defaults off.
• engine.encryptExecutionPayloads is new and defaults off; enabling it uses a payload key under encryption.* that is generated and preserved like the root key.
• Worker egress allowlists need uploads.linear.app, and *.service-now.com and the Mixpanel endpoints if those toolkits are enabled.
• Engine 5xx rates drop as disconnects, malformed gateway IDs, and hook conflicts leave that population; alert thresholds tuned to the old baseline will be loose.
• The seed-idp hook's deadline is now derived from the OpenFGA resolver's own wait budget, so a slow PDP rollout no longer fails the release.

• Scheduled-tools list and detail endpoints: 410 Gone.
• Withdrawn worker-management routes: 405 becomes 410, carrying the migration guide.
• Client disconnects: 500 becomes 499. Malformed gateway IDs: 404. Plugin hook priority conflicts: 409.
• Billing reads require the organization-admin billing-read permission.
• Registering a model through the admin API requires public https.
• Audit events gain a viewed action and a tool_execution resource type.