SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support

The Thesis

Enterprise AI agents fail quietly: they get deployed with generic skills, hit edge cases, and no one fixes the underlying logic. SkillForge proposes a closed loop where an agent not only executes cloud support tasks but automatically diagnoses its own failures and rewrites the skills — the reusable procedural modules it follows — that caused them. Tested on nearly 1,900 real support tickets across five cloud scenarios, the system's auto-evolved skills eventually beat manually written expert procedures. The catch is that 'beating expert knowledge' is measured by consistency with historical reference responses, not by customer satisfaction or resolution rates in live production. Still, for any enterprise running AI-powered helpdesks or IT operations, a framework that compounds quality with every deployment cycle is meaningfully different from one that stays frozen at launch.

Catalyst

Large language models (LLMs) capable of following complex procedural instructions only reached reliable enterprise-grade quality in the last 18–24 months. Simultaneously, cloud support orgs have accumulated large archives of historical tickets with expert-authored resolutions — exactly the training signal SkillForge needs for domain grounding. The confluence of capable foundation models and rich enterprise knowledge archives makes automated skill synthesis tractable now in a way it wasn't when GPT-3-era models struggled with multi-step tool use.

What's New

Prior agent skill creation systems — such as Voyager (used in game environments) and generic tool-synthesis frameworks — create skills from scratch using only the model's general knowledge, with no grounding in the specific domain the agent will actually operate in. Those approaches also treat deployment as a one-way door: once a skill is written, failures accumulate but nothing feeds back to fix the code. SkillForge adds two things: a Domain-Contextualized Skill Creator that mines existing knowledge bases and ticket histories before writing any skill, and a three-stage failure loop (Failure Analyzer → Skill Diagnostician → Skill Optimizer) that iteratively patches deficiencies after each deployment batch.

The Counter

The core evaluation metric — how closely the agent's output matches expert-authored historical responses — is a proxy, not a measure of actual customer outcomes. A skill can be highly consistent with old tickets while still failing on genuinely new problem types that weren't in the training archive. The benchmark covers five scenarios at a single organization; it's entirely unclear whether the self-evolution loop generalizes to other cloud platforms, other support domains, or adversarial edge cases. The paper also doesn't report how many rounds of evolution were needed to surpass expert baselines, or whether quality plateaus or degrades over many cycles. Perhaps most importantly, the system relies on an LLM to diagnose its own failures and rewrite its own skills — a process that could silently introduce new failure modes while fixing the targeted ones. Existing ITSM platforms with strong workflow engines and human-in-the-loop escalation already handle the hard cases reasonably well; the marginal value of full automation depends heavily on a cost-quality tradeoff the paper doesn't quantify.

Resources

• https://arxiv.org/abs/2604.08618
• https://arxiv.org/abs/2305.16291
• https://arxiv.org/abs/2310.05915
• https://en.wikipedia.org/wiki/IT_service_management
• https://en.wikipedia.org/wiki/Large_language_model#Agents

The Paper