Task Skill Package Eval：任务技能包触发与安全实验¶
生成时间：2026-06-15T10:15:05.071333+00:00
数据分类：synthetic_internal_demo。样例为合成任务，用于验证任务技能包的触发、渐进披露、过期包和安全风险，不代表真实团队聊天或客户材料。
实验定位¶
这个实验回答：任务技能包是否会误触发、漏触发、造成上下文膨胀，或者把过期/不安全包带进任务。它验证的是 Ontology 本体层的可执行知识治理，不替代 D3 检索投影、D4 Evidence Connector 或 D5 主动询问。
策略¶
always_load_all：每个任务都加载所有包和引用，模拟最粗暴的 all-in prompt。keyword_title_only：只看包名/标题关键词，模拟没有 manifest 的浅层触发。progressive_manifest：按 task_type、trigger_terms 和 signals 触发，按需加载 references。progressive_manifest_guarded：在 manifest 触发基础上过滤 deprecated、unsafe、无验证和 negative_terms。
汇总结果¶
| 策略 | Precision | Recall | Exact Match | Safe Activation | Passed | False Positive | False Negative | Forbidden | Stale | Unsafe | Avg Tokens | Max Tokens |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| always_load_all | 10% | 100% | 0% | 0% | 0% | 65 | 0 | 18 | 9 | 9 | 14020.0 | 14020 |
| keyword_title_only | 64% | 100% | 67% | 78% | 67% | 4 | 0 | 2 | 0 | 1 | 1657.8 | 4030 |
| progressive_manifest | 47% | 100% | 33% | 33% | 33% | 8 | 0 | 8 | 6 | 0 | 2720.0 | 6610 |
| progressive_manifest_guarded | 100% | 100% | 100% | 100% | 100% | 0 | 0 | 0 | 0 | 0 | 766.7 | 1740 |
样例明细¶
| 策略 | 任务 | 期望 | 实际 | 问题 | Tokens | 通过 |
|---|---|---|---|---|---|---|
| always_load_all | ingest_wechat_l1_l2_reference | harness_knowledge_ingest | harness_knowledge_ingest, harness_site_publish, agentic_search_benchmark, ask_router_review, tool_gateway_safety, project_status_dashboard, broad_team_memory, raw_script_runner | fp:agentic_search_benchmark, fp:ask_router_review, fp:broad_team_memory, fp:harness_site_publish, fp:project_status_dashboard, fp:raw_script_runner, fp:tool_gateway_safety, forbidden:harness_site_publish, forbidden:broad_team_memory, stale:broad_team_memory, unsafe:raw_script_runner, no_verify:broad_team_memory, no_verify:raw_script_runner | 14020 | fail |
| always_load_all | publish_site_nav | harness_site_publish | harness_knowledge_ingest, harness_site_publish, agentic_search_benchmark, ask_router_review, tool_gateway_safety, project_status_dashboard, broad_team_memory, raw_script_runner | fp:agentic_search_benchmark, fp:ask_router_review, fp:broad_team_memory, fp:harness_knowledge_ingest, fp:project_status_dashboard, fp:raw_script_runner, fp:tool_gateway_safety, forbidden:broad_team_memory, stale:broad_team_memory, unsafe:raw_script_runner, no_verify:broad_team_memory, no_verify:raw_script_runner | 14020 | fail |
| always_load_all | agentic_search_option_eval | agentic_search_benchmark | harness_knowledge_ingest, harness_site_publish, agentic_search_benchmark, ask_router_review, tool_gateway_safety, project_status_dashboard, broad_team_memory, raw_script_runner | fp:ask_router_review, fp:broad_team_memory, fp:harness_knowledge_ingest, fp:harness_site_publish, fp:project_status_dashboard, fp:raw_script_runner, fp:tool_gateway_safety, forbidden:harness_site_publish, forbidden:broad_team_memory, stale:broad_team_memory, unsafe:raw_script_runner, no_verify:broad_team_memory, no_verify:raw_script_runner | 14020 | fail |
| always_load_all | ask_router_before_wecom_message | ask_router_review | harness_knowledge_ingest, harness_site_publish, agentic_search_benchmark, ask_router_review, tool_gateway_safety, project_status_dashboard, broad_team_memory, raw_script_runner | fp:agentic_search_benchmark, fp:broad_team_memory, fp:harness_knowledge_ingest, fp:harness_site_publish, fp:project_status_dashboard, fp:raw_script_runner, fp:tool_gateway_safety, forbidden:broad_team_memory, stale:broad_team_memory, unsafe:raw_script_runner, no_verify:broad_team_memory, no_verify:raw_script_runner | 14020 | fail |
| always_load_all | tool_gateway_schema_safety | tool_gateway_safety | harness_knowledge_ingest, harness_site_publish, agentic_search_benchmark, ask_router_review, tool_gateway_safety, project_status_dashboard, broad_team_memory, raw_script_runner | fp:agentic_search_benchmark, fp:ask_router_review, fp:broad_team_memory, fp:harness_knowledge_ingest, fp:harness_site_publish, fp:project_status_dashboard, fp:raw_script_runner, forbidden:broad_team_memory, forbidden:raw_script_runner, stale:broad_team_memory, unsafe:raw_script_runner, no_verify:broad_team_memory, no_verify:raw_script_runner | 14020 | fail |
| always_load_all | casual_obsidian_vs_es_question | - | harness_knowledge_ingest, harness_site_publish, agentic_search_benchmark, ask_router_review, tool_gateway_safety, project_status_dashboard, broad_team_memory, raw_script_runner | fp:agentic_search_benchmark, fp:ask_router_review, fp:broad_team_memory, fp:harness_knowledge_ingest, fp:harness_site_publish, fp:project_status_dashboard, fp:raw_script_runner, fp:tool_gateway_safety, forbidden:harness_site_publish, forbidden:agentic_search_benchmark, forbidden:broad_team_memory, stale:broad_team_memory, unsafe:raw_script_runner, no_verify:broad_team_memory, no_verify:raw_script_runner | 14020 | fail |
| always_load_all | customer_contract_sensitive_source | - | harness_knowledge_ingest, harness_site_publish, agentic_search_benchmark, ask_router_review, tool_gateway_safety, project_status_dashboard, broad_team_memory, raw_script_runner | fp:agentic_search_benchmark, fp:ask_router_review, fp:broad_team_memory, fp:harness_knowledge_ingest, fp:harness_site_publish, fp:project_status_dashboard, fp:raw_script_runner, fp:tool_gateway_safety, forbidden:harness_knowledge_ingest, forbidden:broad_team_memory, forbidden:raw_script_runner, stale:broad_team_memory, unsafe:raw_script_runner, no_verify:broad_team_memory, no_verify:raw_script_runner | 14020 | fail |
| always_load_all | update_project_progress_dashboard | project_status_dashboard | harness_knowledge_ingest, harness_site_publish, agentic_search_benchmark, ask_router_review, tool_gateway_safety, project_status_dashboard, broad_team_memory, raw_script_runner | fp:agentic_search_benchmark, fp:ask_router_review, fp:broad_team_memory, fp:harness_knowledge_ingest, fp:harness_site_publish, fp:raw_script_runner, fp:tool_gateway_safety, forbidden:harness_site_publish, forbidden:broad_team_memory, stale:broad_team_memory, unsafe:raw_script_runner, no_verify:broad_team_memory, no_verify:raw_script_runner | 14020 | fail |
| always_load_all | benchmark_without_publish | agentic_search_benchmark | harness_knowledge_ingest, harness_site_publish, agentic_search_benchmark, ask_router_review, tool_gateway_safety, project_status_dashboard, broad_team_memory, raw_script_runner | fp:ask_router_review, fp:broad_team_memory, fp:harness_knowledge_ingest, fp:harness_site_publish, fp:project_status_dashboard, fp:raw_script_runner, fp:tool_gateway_safety, forbidden:harness_site_publish, forbidden:broad_team_memory, stale:broad_team_memory, unsafe:raw_script_runner, no_verify:broad_team_memory, no_verify:raw_script_runner | 14020 | fail |
| keyword_title_only | ingest_wechat_l1_l2_reference | harness_knowledge_ingest | harness_knowledge_ingest, harness_site_publish, raw_script_runner | fp:harness_site_publish, fp:raw_script_runner, forbidden:harness_site_publish, unsafe:raw_script_runner, no_verify:raw_script_runner | 4030 | fail |
| keyword_title_only | publish_site_nav | harness_site_publish | harness_site_publish | - | 1070 | pass |
| keyword_title_only | agentic_search_option_eval | agentic_search_benchmark | agentic_search_benchmark | - | 1620 | pass |
| keyword_title_only | ask_router_before_wecom_message | ask_router_review | harness_knowledge_ingest, ask_router_review | fp:harness_knowledge_ingest | 2620 | fail |
| keyword_title_only | tool_gateway_schema_safety | tool_gateway_safety | tool_gateway_safety | - | 1740 | pass |
| keyword_title_only | casual_obsidian_vs_es_question | - | - | - | 0 | pass |
| keyword_title_only | customer_contract_sensitive_source | - | - | - | 0 | pass |
| keyword_title_only | update_project_progress_dashboard | project_status_dashboard | harness_site_publish, project_status_dashboard | fp:harness_site_publish, forbidden:harness_site_publish | 2220 | fail |
| keyword_title_only | benchmark_without_publish | agentic_search_benchmark | agentic_search_benchmark | - | 1620 | pass |
| progressive_manifest | ingest_wechat_l1_l2_reference | harness_knowledge_ingest | harness_knowledge_ingest, harness_site_publish, broad_team_memory | fp:broad_team_memory, fp:harness_site_publish, forbidden:harness_site_publish, forbidden:broad_team_memory, stale:broad_team_memory, no_verify:broad_team_memory | 6610 | fail |
| progressive_manifest | publish_site_nav | harness_site_publish | harness_site_publish, broad_team_memory | fp:broad_team_memory, forbidden:broad_team_memory, stale:broad_team_memory, no_verify:broad_team_memory | 5270 | fail |
| progressive_manifest | agentic_search_option_eval | agentic_search_benchmark | agentic_search_benchmark, broad_team_memory | fp:broad_team_memory, forbidden:broad_team_memory, stale:broad_team_memory, no_verify:broad_team_memory | 5820 | fail |
| progressive_manifest | ask_router_before_wecom_message | ask_router_review | ask_router_review | - | 380 | pass |
| progressive_manifest | tool_gateway_schema_safety | tool_gateway_safety | tool_gateway_safety | - | 1740 | pass |
| progressive_manifest | casual_obsidian_vs_es_question | - | broad_team_memory | fp:broad_team_memory, forbidden:broad_team_memory, stale:broad_team_memory, no_verify:broad_team_memory | 1200 | fail |
| progressive_manifest | customer_contract_sensitive_source | - | - | - | 0 | pass |
| progressive_manifest | update_project_progress_dashboard | project_status_dashboard | project_status_dashboard, broad_team_memory | fp:broad_team_memory, forbidden:broad_team_memory, stale:broad_team_memory, no_verify:broad_team_memory | 1530 | fail |
| progressive_manifest | benchmark_without_publish | agentic_search_benchmark | harness_site_publish, agentic_search_benchmark, broad_team_memory | fp:broad_team_memory, fp:harness_site_publish, forbidden:harness_site_publish, forbidden:broad_team_memory, stale:broad_team_memory, no_verify:broad_team_memory | 1930 | fail |
| progressive_manifest_guarded | ingest_wechat_l1_l2_reference | harness_knowledge_ingest | harness_knowledge_ingest | - | 1340 | pass |
| progressive_manifest_guarded | publish_site_nav | harness_site_publish | harness_site_publish | - | 1070 | pass |
| progressive_manifest_guarded | agentic_search_option_eval | agentic_search_benchmark | agentic_search_benchmark | - | 1620 | pass |
| progressive_manifest_guarded | ask_router_before_wecom_message | ask_router_review | ask_router_review | - | 380 | pass |
| progressive_manifest_guarded | tool_gateway_schema_safety | tool_gateway_safety | tool_gateway_safety | - | 1740 | pass |
| progressive_manifest_guarded | casual_obsidian_vs_es_question | - | - | - | 0 | pass |
| progressive_manifest_guarded | customer_contract_sensitive_source | - | - | - | 0 | pass |
| progressive_manifest_guarded | update_project_progress_dashboard | project_status_dashboard | project_status_dashboard | - | 330 | pass |
| progressive_manifest_guarded | benchmark_without_publish | agentic_search_benchmark | agentic_search_benchmark | - | 420 | pass |
当前结论¶
progressive_manifest_guarded在当前样例中 Precision、Recall、Exact Match 和 Safe Activation 都是 100%，说明任务技能包可以作为 Ontology 本体层的可执行知识对象，但必须有 manifest、negative_terms、验证和安全过滤。always_load_all虽然 Recall 是 100%，但 false positive、forbidden、stale 和 unsafe 都很高，平均上下文 token 是 14020.0。这验证了不能把所有技能包常驻进 prompt。- 渐进披露后的平均上下文 token 比 all-in 方案降低约 95%，同时没有牺牲召回。这个收益来自入口小、引用按需读，而不是来自模型能力提升。
- 单靠关键词标题会漏掉中文语义和工作流意图，也会被“ES/知识/文档”等泛词误导；正式包必须有 task_type、trigger_terms、negative_terms、status、verify 和安全字段。
- 任务技能包不能承载事实源。它只规范 Agent 如何做任务；事实仍要回到 knowledge/site、SearchConnector、源系统和人工 review。
下一步¶
- 把任务包使用日志接入 dashboard：触发次数、误触发、漏触发、平均 tokens、人工纠正。
- 用真实任务日志替换这组合成样例。
ask_router_review高风险包已实体化；后续把真实问错人、节流、回复解析和 knowledge card review 纠正信号写回 usage dashboard。tool_gateway_safety已完成第一轮，后续纳入 usage dashboard 和真实 Connector 接入验证。- 后续接入企微后，把任务包触发和主动询问流程联动。