Forecast Validation — Palm vs Kyriba¶
Metadata¶
- Customer: ON
- Status: Active (ongoing methodology)
- Domain(s): Cash Forecasting, Variance Analysis, Categorization
- Started: 2025-12 (W51 2025)
- Source: FC Validation Deck
Problem Statement¶
ON runs parallel forecast models in Palm (ML-based) and Kyriba (deterministic, invoice-driven). The treasury R&A team needed a rigorous, data-driven framework to validate which model performs better and should drive the new cash planning ecosystem.
Key question: Why invest in ML when Kyriba already has forecasting?
- Kyriba relies on deterministic rules (invoice due dates) — only shows what is already booked, misses unbooked/last-minute transactions
- Palm uses ML to learn cash flow rhythms from historical behavior — dynamic, adaptive predictions
- Palm uses AI to actively re-categorize transactions; Kyriba requires extensive manual rule mapping
Solution / Approach¶
Variance Analysis Methodology¶
Primary KPI: WMAPE (Weighted Mean Absolute Percentage Error)
WMAPE weights the error by the volume of cash compared to the global level. A 10% error on a massive account (like On Inc.) impacts the score more than a 50% error on a dormant entity.
Two measurement dimensions:
- Snapshot Accuracy — WMAPE across all weeks within a single 13-week forecast snapshot. Tracks WoW improvements tied to re-categorization efforts.
- Horizon Accuracy — WMAPE across all snapshots when looking X weeks ahead. Shows how performance degrades with forecast distance.
Analysis Structure¶
- Time horizon: 7-week focused evaluation window (W51 2025 to W5 2026), extracted from each 13-week forecast
- Validation cycle: Forecasts captured at week t, measured against actuals at t+1
- Granularity: Global Operating Cash Position, weekly (every Tuesday)
Measured Outcomes¶
| Metric | Palm | Kyriba | Date Measured |
|---|---|---|---|
| Avg WMAPE (latest snapshot W5) | 5.0% | 8.6% | 2026-01-28 |
| Avg WMAPE (initial W51) | 11.4% | 29.4% | 2025-12-17 |
| 1-week ahead accuracy | 4% | 7% | Avg across all snapshots |
| 2-week ahead accuracy | 5% | 18% | Avg across all snapshots |
| WMAPE improvement trend | 11.4% → 5.0% | 29.4% → 8.6% | W51 to W5 |
| CapEx categorization precision | 96% (up from 90%) | — | 2026-02 |
| CapEx targeted vendor recall | 44% (up from 8.9%) | — | 2026-02 |
| CapEx vendor precision | 100% (up from 95.3%) | — | 2026-02 |
WMAPE by Forecast Snapshot¶
| Analysis Week | Avg WMAPE Palm | Avg WMAPE Kyriba | Difference |
|---|---|---|---|
| 2025.51 | 11.4% | 29.4% | -18% |
| 2025.52 | 8.9% | 20.5% | -12% |
| 2026.2 | 8.4% | 11.5% | -3% |
| 2026.3 | 8.7% | 12.1% | -3% |
| 2026.4 | 6.4% | 13.6% | -7% |
| 2026.5 | 5.0% | 8.6% | -4% |
WMAPE by Weeks Ahead¶
| Weeks Ahead | Palm | Kyriba | Difference |
|---|---|---|---|
| 1 | 4% | 7% | -4% |
| 2 | 5% | 18% | -13% |
| 3 | 8% | 12% | -5% |
| 4 | 9% | 20% | -11% |
| 5 | 18% | 31% | -13% |
| 6 | 23% | 50% | -27% |
| 7 | 17% | 38% | -21% |
Key Findings¶
Strategic Comparison¶
| Criteria | Kyriba | Palm |
|---|---|---|
| Forecast method | Starting Balance + AR-AP Data. Linear trends that "miss" data over time due to 30-60 day payment terms. High volatility from AR invoice concentration. | Time Series Analysis. Captures cash flow rhythms. Low volatility. Theoretically infinite horizon since it doesn't require booked data. |
| Maintenance | Highly dependent on invoices, postings booked into D365, and manual updates (batching rules). | Successfully absorbed mass re-categorization quickly, driving sequential reduction in global variance down to 5%. |
| Trend behavior | Linear FC based on manual inputs and rules. | Reverts to mean (safer forecast). |
Conclusion¶
Palm is the definitive choice to drive the new cash planning ecosystem. It provides a stable, single-digit risk environment on the short term. Although Palm lacks native ownership of AP/AR subledgers and depends on BigQuery integrations and strict data categorization, its behavioral ML engine outperforms over the 7-week span.
ON Team Learnings¶
Categorization¶
- Even with prompting, transaction categorization required significant manual effort initially
- Relied heavily on bank account-based mapping, requiring frequent adjustments
- APAC entities were more complex due to language differences and symbol formats — needs additional regional support
Visibility¶
- Challenges separating operating cash from investment flows in Palm — had to manually exclude investment cash inflows
- Forecasts in Palm were "frozen" for longer-term weeks (8-13) — resolved in close contact with Palm team
KPIs (refining)¶
- Minimum Cash Level
- 4-Week Rolling Average WMAPE
- Global Total WMAPE (with quarterly reduction target)
- Forecast Bias % (systematic over/under forecasting)
- WMAPE by Category, Entity
E2E Cash Forecasting Ecosystem¶
ON is building an integrated liquidity ecosystem across three tools:
| Tool | Horizon | Role |
|---|---|---|
| Kyriba | Actuals | Starting point, centralization, source of truth for bank balances |
| Palm | 13 weeks | Tactical/operational short-term cash management insights |
| Anaplan | Full year | Long-term LRP, investment and capital allocation insights |
Palm's 13-week forecast feeds the rolling quarterly view. Anaplan connects cash to long-term budget plans. Kyriba provides the real-time actuals baseline.
Next Steps — Palm x ON Roadmap 2026¶
| Quarter | Challenge | Action |
|---|---|---|
| Q1 2026 | Categorization errors + IC flow forecast limitations | Track categorization accuracy via self-reporting, bank account category mapping in-tool, new IC and cash pool forecast models |
| Q1 2026 | Ad hoc reporting for cash management | Palm Chat for daily/weekly insights with Slack integration |
| Q2 2026 | Manual variance analysis data storage | Palm "in house" VA reporting with visuals + KPIs (bias, improvements) |
| Q2 2026 | Historical bank statements as only data source | Palm to pull AP/AR data via BigQuery integration; scenario planning for growth assumptions |
| Q3 2026 | Forecast accuracy decrease W5-W13 | New Palm forecasting architecture for higher accuracy |
2026+ Ambitions¶
- AI insights on cash concentration optimization
- FX hedge vs. forecast tracking to identify open FX positions
- Expand forecast beyond 13 weeks
- Activate Palm Chat via Gemini for forecasting, recategorization, and analysis actions