diff --git a/.agents/skills/add-feature/SKILL.md b/.agents/skills/add-feature/SKILL.md new file mode 100644 index 00000000..2873e72c --- /dev/null +++ b/.agents/skills/add-feature/SKILL.md @@ -0,0 +1,284 @@ +--- +name: add-feature +description: Add new T-SQL features to ScriptDOM parser. Asks questions about SQL Server version and feature type, then guides through grammar changes, AST updates, script generation, and testing using existing instruction files. Handles syntax additions, new functions, data types, index types, and validation rules. +argument-hint: "Feature description (e.g., 'Add VECTOR data type', 'Add JSON_OBJECT function')" +user-invocable: true +platforms: "SQL Server, Azure SQL DB, Fabric, Fabric DW, VNext" +--- + +## Overview + +This skill helps add new T-SQL features to the ScriptDOM parser by: +1. **Interviewing** you about the feature and target platform (SQL Server, Azure SQL DB, Fabric, Fabric DW, VNext) +2. **Determining the latest parser version** from `SqlVersionFlags.cs` +3. **Routing** you to the appropriate parser: + - SQL Server → version-specific parser (TSql120-180) + - Azure SQL DB/Fabric/VNext → latest parser version + - Fabric DW → separate TSqlFabricDW parser +4. **Classifying** the change type (grammar, validation, function, data type, etc.) +5. **Guiding** through implementation and testing without duplicating existing documentation + +--- + +## Step 1: Feature Discovery Interview + +Ask the user these questions to understand the feature: + +### Required Questions +1. **What T-SQL feature are you adding?** + - Example: "VECTOR data type", "JSON_OBJECT function", "RESUMABLE option for ALTER TABLE" + +2. **Which platform is this feature for?** + - **SQL Server** (on-premises / standalone) + - **Azure SQL Database** (uses latest parser version) + - **Fabric** (uses latest parser version) + - **Fabric DW** (uses separate TSqlFabricDW parser) + - **VNext** (future version, uses latest parser version unless repo instructions say otherwise) + +3. **If SQL Server, which version introduced this feature?** + - SQL Server 2014 (TSql120) + - SQL Server 2016 (TSql130) + - SQL Server 2017 (TSql140) + - SQL Server 2019 (TSql150) + - SQL Server 2022 (TSql160) + - SQL Server 2025 (TSql170) + - SQL Server vNext (TSql180) + - *(Skip if Azure SQL DB, Fabric, or VNext - these use TSql180 by default)* + +4. **Do you have example T-SQL syntax or Microsoft documentation links?** + - This helps verify the exact syntax requirements + +### Optional Context Questions +4. **Is this a new syntax element or enabling existing syntax in a new context?** + - New syntax: Requires grammar rules, AST nodes + - Validation fix: May only need version checks in validation code + +5. **Does similar syntax already work in other contexts?** + - Example: "RESUMABLE works for ALTER INDEX but not ALTER TABLE" + - This indicates a validation issue rather than missing grammar + +--- + +## Step 2: Determine Feature Type + +Based on the answers, classify the feature into one of these types: + +### Type A: Validation-Only Fix +**Indicators:** +- Similar syntax already works elsewhere in SQL +- Error message says "Option 'X' is not valid..." or "Feature 'Y' not supported..." +- The grammar may already parse it, but validation blocks it + +**→ Route to:** [grammar_validation.guidelines.instructions.md](../../instructions/grammar_validation.guidelines.instructions.md) + +### Type B: New Grammar/Syntax +**Indicators:** +- Completely new statement, clause, or operator +- Error says "Incorrect syntax near..." or "Unexpected token..." +- Parser doesn't recognize the syntax at all + +**→ Route to:** [bug_fixing.guidelines.instructions.md](../../instructions/bug_fixing.guidelines.instructions.md) + +### Type C: New System Function +**Indicators:** +- Adding a new built-in T-SQL function (e.g., JSON_OBJECT, STRING_AGG) +- May need special handling for RETURN statement contexts + +**→ Route to:** [function.guidelines.instructions.md](../../instructions/function.guidelines.instructions.md) + +### Type D: New Data Type +**Indicators:** +- Adding a completely new SQL Server data type (e.g., VECTOR, GEOGRAPHY) +- Requires custom parameter syntax (e.g., `VECTOR(1536, FLOAT32)`) + +**→ Route to:** [new_data_types.guidelines.instructions.md](../../instructions/new_data_types.guidelines.instructions.md) + +### Type E: New Index Type +**Indicators:** +- Adding a new CREATE INDEX variant (e.g., VECTOR INDEX, JSON INDEX) +- Requires specialized syntax different from standard indexes + +**→ Route to:** [new_index_types.guidelines.instructions.md](../../instructions/new_index_types.guidelines.instructions.md) + +### Type F: Parser Predicate Recognition (Parentheses) +**Indicators:** +- Identifier-based predicate works without parentheses but fails with them +- Error near closing parenthesis: `WHERE (REGEXP_LIKE(...))` fails + +**→ Route to:** [parser.guidelines.instructions.md](../../instructions/parser.guidelines.instructions.md) + +### Type G: Database Option (ALTER/CREATE DATABASE SET) +**Indicators:** +- Adding a database option for ALTER DATABASE or CREATE DATABASE statements +- Simple ON/OFF option (e.g., `SET AUTOMATIC_INDEX_COMPACTION = ON`) +- Complex option with sub-options (e.g., `SET CHANGE_TRACKING (AUTO_CLEANUP = ON)`) +- Enum-based option (e.g., `SET RECOVERY FULL`) + +**→ Route to:** [database_option.guidelines.instructions.md](../../instructions/database_option.guidelines.instructions.md) + +--- + +## Step 3: Grammar Rule Development + +For grammar changes (Types B, C, D, E, G), follow this workflow: + +### 3.1 Identify Target Grammar File + +**First, determine the latest parser version:** +Check `SqlScriptDom/Parser/TSql/SqlVersionFlags.cs` for the highest TSql version enum value. + +**Then select the appropriate grammar file:** + +#### For SQL Server (on-premises): +- SQL 2014+: TSql120.g +- SQL 2016+: TSql130.g +- SQL 2017+: TSql140.g +- SQL 2019+: TSql150.g +- SQL 2022+: TSql160.g +- SQL 2025+: TSql170.g +- SQL Server vNext: TSql180.g + +#### For Azure SQL Database, Fabric, VNext: +- Use the **latest parser version** (currently TSql180.g) +- Check `SqlVersionFlags.cs` to confirm the highest version + +#### For Fabric DW: +- Use **TSqlFabricDW.g** (separate parser with Fabric-specific syntax) + +### 3.2 Grammar Development Patterns +**Reference:** [grammar.guidelines.instructions.md](../../instructions/grammar.guidelines.instructions.md) + +Key patterns: +- Use `this.FragmentFactory.CreateFragment()` for AST nodes +- Use syntactic predicates `(LA(1) == Token)` for lookahead +- Avoid modifying shared grammar rules (create specialized versions instead) +- Include proper script generation in `ScriptGenerator` + +### 3.3 AST Updates +If adding new AST nodes: +1. Edit `SqlScriptDom/Parser/TSql/Ast.xml` +2. Rebuild to regenerate visitor classes +3. Implement script generation in `SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator` + +--- + +## Step 4: Testing + +**Reference:** [testing.guidelines.instructions.md](../../instructions/testing.guidelines.instructions.md) + +### 4.1 Create Test Files +1. **Input Script**: `Test/SqlDom/TestScripts/YourFeature.sql` + - Include all syntax variations + - Test edge cases, parameters, expressions + +2. **Baseline**: `Test/SqlDom/Baselines/YourFeature.sql` + - Expected pretty-printed output after parse → script gen → reparse + +### 4.2 Add Test Method + +#### For SQL Server versions (TSql120-180): +In `Test/SqlDom/OnlySyntaxTests.cs`: + +```csharp +[TestMethod] +public void YourFeatureTest() +{ + ParserTest("YourFeature.sql"); +} +``` + +#### For Fabric DW: +Add to the `OnlyFabricDWTestInfos` array in `Test/SqlDom/OnlyFabricDWSyntaxTests.cs`: + +```csharp +new ParserTestFabricDW("YourFeatureFabricDW.sql", + nErrors80: X, nErrors90: Y, nErrors100: Z, + nErrors110: A, nErrors120: B, nErrors130: C, + nErrors140: D, nErrors150: E, nErrors160: F, nErrors170: G, nErrors180: H), +``` +*(Set expected error counts for each parser version)* + +**Simplified patterns available** - see testing guidelines for: +- `ParserTestAllowingErrors` - for scripts with expected parse errors +- `ParserTestOutput` constructors - for custom output verification +- Error message verification - exact match or fragment-based + +### 4.3 Run Tests +```bash +# Build to regenerate parser +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "FullyQualifiedName~YourFeatureTest" -c Debug + +# ALWAYS run full suite before committing +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +--- + +## Step 5: Debugging and Troubleshooting + +**Reference:** [debugging_workflow.guidelines.instructions.md](../../instructions/debugging_workflow.guidelines.instructions.md) + +Common issues: +- **Tests fail after grammar changes**: Likely modified a shared rule - create specialized version +- **Script generation mismatch**: Update `ScriptGenerator` for your AST nodes +- **Version errors**: Check validation code in `TSql80ParserBaseInternal.cs` + +--- + +## Step 6: Implementation Checklist + +Before marking the feature complete: + +- [ ] Grammar rules added/modified in correct `TSql.g` or `TSqlFabricDW.g` file +- [ ] AST nodes defined in `Ast.xml` (if new nodes needed) +- [ ] Script generation implemented for new AST nodes +- [ ] Validation rules updated (if version-gated) +- [ ] Test script created in `TestScripts/` (use appropriate suffix: `.sql` or `FabricDW.sql`) +- [ ] Baseline created in `Baselines/` or `BaselinesFabricDW/` +- [ ] Test method added to `OnlySyntaxTests.cs` or `OnlyFabricDWSyntaxTests.cs` +- [ ] All tests pass (including full test suite) +- [ ] No regressions in unrelated tests + +--- + +## Quick Reference: File Locations + +| Component | Path | +|-----------|------| +| Grammar files (SQL Server) | `SqlScriptDom/Parser/TSql/TSql*.g` | +| Grammar file (Fabric DW) | `SqlScriptDom/Parser/TSql/TSqlFabricDW.g` | +| Version detection | `SqlScriptDom/Parser/TSql/SqlVersionFlags.cs` | +| AST definition | `SqlScriptDom/Parser/TSql/Ast.xml` | +| Script generator | `SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator` | +| Validation code | `SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs` | +| Test scripts | `Test/SqlDom/TestScripts/` | +| Baselines | `Test/SqlDom/Baselines/` or `BaselinesFabricDW/` | +| Test classes | `Test/SqlDom/OnlySyntaxTests.cs` or `FabricDWSyntaxTests.cs` | + +--- + +## Agent Workflow + +When invoked, the agent should: + +1. **Determine the latest parser version** by reading `SqlScriptDom/Parser/TSql/SqlVersionFlags.cs` +2. **Interview the user** using the questions in Step 1 + - Ask about platform (SQL Server, Azure SQL DB, Fabric, Fabric DW, VNext) + - For Azure/Fabric/VNext → use latest parser version + - For Fabric DW → use TSqlFabricDW.g + - For SQL Server → ask for specific version +3. **Classify the feature type** using Step 2 criteria +4. **Load the appropriate instruction file** using `read_file` tool +5. **Guide through implementation** by referencing specific sections of the loaded instruction file +6. **Execute the development workflow**: + - Make grammar/validation changes + - Regenerate parser (build) + - Create test files + - Run tests + - Debug failures +7. **Verify completeness** using Step 6 checklist + +**Never duplicate content from instruction files** - always reference and quote specific sections. diff --git a/.agents/skills/verify-and-test-tsql-syntax/SKILL.md b/.agents/skills/verify-and-test-tsql-syntax/SKILL.md new file mode 100644 index 00000000..fed8121f --- /dev/null +++ b/.agents/skills/verify-and-test-tsql-syntax/SKILL.md @@ -0,0 +1,115 @@ +--- +name: verify-and-test-tsql-syntax +description: Verify whether an exact T-SQL script is already supported by SqlScriptDOM, then add or update the correct parser tests and baselines. Use when a user asks if syntax already works, or when adding regression coverage for a specific script. +argument-hint: "Exact T-SQL script or syntax to verify" +user-invocable: true +--- + +## Overview + +This skill determines whether a specific T-SQL script is already supported and then guides the follow-up testing work. + +Use it for: +- verifying whether a script already parses +- confirming which parser version should support the syntax +- adding or updating regression coverage in the existing ScriptDOM test framework + +Do not use it for broad feature design. If the verification shows missing parser support, route the implementation work to the appropriate repo instruction: +- [bug_fixing.guidelines.instructions.md](../../../.github/instructions/bug_fixing.guidelines.instructions.md) +- [grammar_validation.guidelines.instructions.md](../../../.github/instructions/grammar_validation.guidelines.instructions.md) +- [parser.guidelines.instructions.md](../../../.github/instructions/parser.guidelines.instructions.md) +- [testing.guidelines.instructions.md](../../../.github/instructions/testing.guidelines.instructions.md) + +## Core Rules + +- Always verify the exact T-SQL text first, character for character. +- For first-pass verification, add a debug unit test method to an existing test file such as `Only180SyntaxTests.cs` when the syntax targets vNext or the latest parser. +- Do not create standalone console apps, ad hoc parsers, or extra test projects. +- After verification, add proper test coverage through `TestScripts`, baselines, and the existing syntax test classes. +- Remove any temporary debug-only test methods once the result is confirmed. + +## Workflow + +### 1. Gather the minimum inputs + +Ask for: +- the exact T-SQL script to verify +- the SQL Server version or parser version expected to support it +- the current behavior and the expected behavior + +If the user has not supplied the exact script yet, stop and request it before continuing. + +### 2. Verify the exact script first + +Add a temporary debug unit test method to the appropriate existing test class and test the exact script as provided. + +Use this sequence: + +```bash +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug +dotnet test --filter "DebugExactScriptTest" Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +Verification expectations: +- If parsing succeeds, move directly to comprehensive test coverage. +- If parsing fails, capture the exact parse errors and classify the failure. + +### 3. Classify the gap if verification fails + +Use the failure mode to route the next step: +- Grammar issue: parser does not recognize the syntax at all. +- Validation issue: syntax parses but is rejected as unsupported or invalid in context. +- Predicate recognition issue: identifier-based predicate fails in parentheses. + +When similar syntax already works in another context, prefer validation analysis before changing grammar. + +### 4. Add comprehensive coverage + +After verification, create or update the proper permanent tests using the existing framework: + +- Add the input script under `Test/SqlDom/TestScripts/`. +- Add the matching baseline under `Test/SqlDom/Baselines/`. +- Add the `ParserTest` entry or permanent test method in the correct `OnlySyntaxTests.cs` file. +- Preserve the exact user syntax in the test script before adding broader coverage. + +Determine the parser version using the repo mapping: +- SQL Server 2014 -> TSql120 +- SQL Server 2016 -> TSql130 +- SQL Server 2017 -> TSql140 +- SQL Server 2019 -> TSql150 +- SQL Server 2022 -> TSql160 +- SQL Server 2025 -> TSql170 +- SQL Server vNext/latest -> TSql180 + +### 5. Validate the testing slice + +Run validation in this order: + +```bash +dotnet test --filter "FullyQualifiedName~YourFeatureTest" Test/SqlDom/UTSqlScriptDom.csproj -c Debug +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +If the targeted test fails because of a baseline mismatch: +- use the generated output from the failure log to update the baseline +- rerun the same targeted test +- only then run the full suite + +## File Targets + +Use these repo paths when doing the work: +- Grammar files: `SqlScriptDom/Parser/TSql/TSql*.g` +- AST definition: `SqlScriptDom/Parser/TSql/Ast.xml` +- Validation code: `SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs` +- Test scripts: `Test/SqlDom/TestScripts/` +- Baselines: `Test/SqlDom/Baselines/` +- Syntax tests: `Test/SqlDom/OnlySyntaxTests.cs` + +## Completion Checklist + +- [ ] Verified the exact script first in an existing test file +- [ ] Classified the result as already supported or missing support +- [ ] Added or updated permanent test coverage in the standard test framework +- [ ] Preserved the exact user syntax in the new test script +- [ ] Ran targeted validation for the touched test slice +- [ ] Ran the full `UTSqlScriptDom` suite before finishing \ No newline at end of file diff --git a/.github/README.md b/.github/README.md new file mode 100644 index 00000000..ce0ce578 --- /dev/null +++ b/.github/README.md @@ -0,0 +1,304 @@ +# SqlScriptDOM Documentation Guide + +Welcome to the SqlScriptDOM documentation! This folder contains comprehensive guides for understanding, developing, and debugging the SQL Server T-SQL parser. + +## 🚀 Quick Start + +**New to the project?** Start here: +1. Read [copilot-instructions.md](copilot-instructions.md) - Main project documentation +2. Browse [debugging_workflow.guidelines.instructions.md](instructions/debugging_workflow.guidelines.instructions.md) - Visual quick reference + +**Fixing a bug?** Start here: +1. Open [debugging_workflow.guidelines.instructions.md](instructions/debugging_workflow.guidelines.instructions.md) - Identify bug type +2. Follow the flowchart to the appropriate guide +3. Use the step-by-step instructions + +## 📚 Documentation Map + +### Core Documentation + +#### [copilot-instructions.md](copilot-instructions.md) - **START HERE** +**Purpose**: Main project documentation and overview +**Contains**: +- Project structure and key files +- Build and test commands +- Developer workflow +- Bug fixing triage +- Debugging tips +- Grammar gotchas and pitfalls + +**When to read**: First time working on the project, or for general context + +--- + +### Quick Reference + +#### [debugging_workflow.guidelines.instructions.md](instructions/debugging_workflow.guidelines.instructions.md) - **QUICK REFERENCE** +**Purpose**: Visual guide for quick bug diagnosis +**Contains**: +- Diagnostic flowchart +- Error pattern recognition +- Investigation steps +- Testing commands reference +- Key files reference +- Common pitfalls + +**When to use**: When you have a bug and need to quickly identify what type of fix is needed + +--- + +### Specialized Fix Guides + +#### [Validation_fix.guidelines.instructions.md](instructions/Validation_fix.guidelines.instructions.md) - Most Common Fix Type ⭐ +**Purpose**: Fixing validation-based bugs +**When to use**: +- ✅ Error: "Option 'X' is not valid..." or "Feature not supported..." +- ✅ Same syntax works in different context (e.g., ALTER INDEX vs ALTER TABLE) +- ✅ SQL Server version-specific features + +**Contains**: +- Real-world example (ALTER TABLE ADD CONSTRAINT RESUMABLE) +- Version flag patterns +- Validation logic modification +- Testing strategy + +**Complexity**: ⭐ Easy +**Typical time**: 1-2 hours + +--- + +#### [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) - Grammar Changes +**Purpose**: Adding new syntax or modifying parser grammar +**When to use**: +- ✅ Error: "Incorrect syntax near..." or "Unexpected token..." +- ✅ Parser doesn't recognize new T-SQL features +- ✅ Need to add new keywords, operators, or statements + +**Contains**: +- Complete bug-fixing workflow +- Grammar modification process +- AST updates +- Script generator changes +- Baseline generation +- Decision tree for bug types + +**Complexity**: ⭐⭐⭐ Medium to Hard +**Typical time**: 4-8 hours + +--- + +#### [parser.guidelines.instructions.md](instructions/parser.guidelines.instructions.md) +**Purpose**: Fixing parentheses recognition issues +**When to use**: +- ✅ `WHERE PREDICATE(...)` works +- ❌ `WHERE (PREDICATE(...))` fails with syntax error +- ✅ Identifier-based boolean predicates + +**Contains**: +- `IsNextRuleBooleanParenthesis()` modification +- Predicate detection patterns +- Real example (REGEXP_LIKE) + +**Complexity**: ⭐⭐ Easy-Medium +**Typical time**: 1-3 hours + +--- + +#### [grammer.guidelines.instructions.md](instructions/grammer.guidelines.instructions.md) +**Purpose**: Common patterns for extending existing grammar +**When to use**: +- ✅ Need to extend literal types to accept expressions +- ✅ Adding new enum members +- ✅ Creating new function/statement types + +**Contains**: +- Literal to expression pattern +- Real example (VECTOR_SEARCH TOP_N) +- Context-specific grammar rules +- Shared rule warnings + +**Complexity**: ⭐⭐⭐ Medium +**Typical time**: 3-6 hours + +--- + +### Meta Documentation + +#### [documentation.guidelines.instructions.md](instructions/documentation.guidelines.instructions.md) +**Purpose**: Summary of documentation improvements +**Contains**: +- What was improved and why +- Before/after comparison +- Real-world validation (ALTER TABLE RESUMABLE) +- Lessons learned + +**When to read**: If you want to understand the documentation structure and evolution + +--- + +## 🎯 Bug Type Decision Tree + +``` +┌─────────────────────────────────┐ +│ You have a parsing bug │ +└───────────┬─────────────────────┘ + │ + ▼ + ┌───────────────┐ + │ What's the │ + │ error message?│ + └───────┬───────┘ + │ + ┌────────┼────────┐ + │ │ │ + ▼ ▼ ▼ +┌──────┐ ┌──────┐ ┌──────┐ +│Option│ │Syntax│ │Parens│ +│error │ │error │ │break │ +└──┬───┘ └──┬───┘ └──┬───┘ + │ │ │ + ▼ ▼ ▼ +┌──────┐ ┌──────┐ ┌──────┐ +│VALID-│ │BUG │ │PARSER│ +│ATION │ │FIXING│ │PRED │ +│FIX │ │GUIDE │ │RECOG │ +└──────┘ └──────┘ └──────┘ +``` + +## 📋 Quick Reference Table + +| Error Message | Bug Type | Guide | Complexity | +|--------------|----------|-------|------------| +| "Option 'X' is not valid in statement Y" | Validation | [Validation_fix.guidelines.instructions.md](instructions/Validation_fix.guidelines.instructions.md) | ⭐ Easy | +| "Feature 'X' not supported in version Y" | Validation | [Validation_fix.guidelines.instructions.md](instructions/Validation_fix.guidelines.instructions.md) | ⭐ Easy | +| "Incorrect syntax near keyword" | Grammar | [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) | ⭐⭐⭐ Medium | +| "Unexpected token" | Grammar | [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) | ⭐⭐⭐ Medium | +| Syntax error with parentheses only | Predicate Recognition | [parser.guidelines.instructions.md](instructions/parser.guidelines.instructions.md) | ⭐⭐ Easy-Medium | +| Need to extend literal to expression | Grammar Extension | [GRAMMAR_EXTENSION_PATTERNS](GRAMMAR_EXTENSION_PATTERNS.md) | ⭐⭐⭐ Medium | + +## 🔍 Common Scenarios + +### Scenario 1: New SQL Server Feature Not Recognized +**Example**: `ALTER TABLE ... WITH (RESUMABLE = ON)` fails +**Likely Issue**: Validation blocking the option +**Start With**: [VALIDATION_FIX_GUIDE.md](VALIDATION_FIX_GUIDE.md) + +### Scenario 2: New T-SQL Keyword Not Parsed +**Example**: `CREATE EXTERNAL TABLE` not recognized +**Likely Issue**: Grammar doesn't have rules for this syntax +**Start With**: [BUG_FIXING_GUIDE.md](BUG_FIXING_GUIDE.md) + +### Scenario 3: Function Works Sometimes, Fails with Parentheses +**Example**: `WHERE REGEXP_LIKE(...)` fails +**Likely Issue**: Predicate recognition +**Start With**: [PARSER_PREDICATE_RECOGNITION_FIX.md](PARSER_PREDICATE_RECOGNITION_FIX.md) + +### Scenario 4: Parameter Support Needed +**Example**: `TOP_N = @parameter` should work +**Likely Issue**: Need to extend from literal to expression +**Start With**: [GRAMMAR_EXTENSION_PATTERNS.md](GRAMMAR_EXTENSION_PATTERNS.md) + +## 🛠️ Essential Commands + +```bash +# Build parser +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "FullyQualifiedName~YourTest" -c Debug + +# Run ALL tests (CRITICAL before committing!) +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Search for error code +grep -r "SQL46057" SqlScriptDom/ + +# Search for option usage +grep -r "RESUMABLE" Test/SqlDom/TestScripts/ +``` + +## 📊 Documentation Statistics + +- **Total Guides**: 6 comprehensive guides +- **Bug Types Covered**: 3 main types (validation, grammar, predicate recognition) +- **Real-World Examples**: 4 detailed examples with code +- **Code Samples**: 50+ practical bash/C#/SQL examples +- **Quick References**: 3 tables and 2 flowcharts + +## 🎓 Learning Path + +### Beginner Path (Understanding the Project) +1. [copilot-instructions.md](copilot-instructions.md) - Read "Key points" section +2. [debugging_workflow.guidelines.instructions.md](instructions/debugging_workflow.guidelines.instructions.md) - Understand bug types +3. [Validation_fix.guidelines.instructions.md](instructions/Validation_fix.guidelines.instructions.md) - Follow ALTER TABLE RESUMABLE example +4. Try fixing a validation bug yourself + +**Time**: 2-3 hours + +### Intermediate Path (Grammar Changes) +1. Review beginner path first +2. [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) - Complete workflow +3. [grammer.guidelines.instructions.md](instructions/grammer.guidelines.instructions.md) - Common patterns +4. [copilot-instructions.md](copilot-instructions.md) - "Grammar Gotchas" section +5. Try adding a simple new keyword + +**Time**: 4-6 hours + +### Advanced Path (Complex Features) +1. Master beginner and intermediate paths +2. [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md) - AST modifications +3. [grammer.guidelines.instructions.md](instructions/grammer.guidelines.instructions.md) - All patterns +4. Study existing complex features (e.g., VECTOR_SEARCH) +5. Implement a new statement type + +**Time**: 8-16 hours + +## 🚨 Critical Reminders + +### Always Do This: +- ✅ **Run full test suite** before committing (1,100+ tests) +- ✅ **Check Microsoft docs** for exact version support +- ✅ **Search for error messages** first before coding +- ✅ **Create context-specific rules** instead of modifying shared ones +- ✅ **Test across all SQL Server versions** in test configuration + +### Never Do This: +- ❌ Modify shared grammar rules without understanding impact +- ❌ Skip running the full test suite +- ❌ Assume version support - always verify documentation +- ❌ Edit generated files in `obj/` directory +- ❌ Commit without testing baseline generation + +## 🤝 Contributing + +When improving these docs: +1. Use real examples from actual bugs +2. Include complete code samples (not pseudo-code) +3. Add bash commands that actually work +4. Cross-reference related guides +5. Update this README if adding new guides + +## 📞 Getting Help + +If stuck: +1. Search error message in codebase: `grep -r "your error"` +2. Check similar working syntax: `grep -r "keyword" Test/SqlDom/` +3. Review relevant guide based on bug type +4. Check Git history for similar fixes: `git log --grep="RESUMABLE"` + +## 🎉 Success Metrics + +You know you've succeeded when: +- ✅ Your specific test passes +- ✅ **ALL 1,100+ tests pass** (critical!) +- ✅ Baseline matches generated output +- ✅ Version-specific behavior is correct +- ✅ No regressions in existing functionality + +--- + +**Last Updated**: Based on ALTER TABLE RESUMABLE fix (October 2025) + +**Contributors**: Documentation improved based on practical bug-fixing experience + +**Feedback**: These guides are living documents. Please update them when you discover new patterns or better approaches! diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 3baaa077..afc4929c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -26,15 +26,21 @@ ScriptDom is a library for parsing and generating T-SQL scripts. It is primarily - `SqlScriptDom/ParserPostProcessing.sed`, `LexerPostProcessing.sed`, `TSqlTokenTypes.ps1` — post-processing for generated C# sources and tokens. - `tools/` — contains code generators used during build: `AstGen`, `ScriptGenSettingsGenerator`, `TokenListGenerator`. - `Test/SqlDom/` — unit tests, baselines and test scripts. See `Only170SyntaxTests.cs`, `TestScripts/`, and `Baselines170/`. +- `.github/instructions/testing.guidelines.instructions.md` — comprehensive testing framework guide with patterns and best practices. +- `.github/instructions/function.guidelines.instructions.md` — specialized guide for adding new T-SQL system functions. +- `.github/instructions/database_option.guidelines.instructions.md` — specialized guide for adding database options (ALTER/CREATE DATABASE SET options). ## Developer workflow & conventions (typical change cycle) 1. Add/modify grammar rule(s) in the correct `TSql*.g` (pick the _version_ the syntax belongs to). 2. If tokens or token ordering change, update `TSqlTokenTypes.g` (and the sed/ps1 post-processors if necessary). 3. Rebuild the ScriptDom project to regenerate parser and AST (`dotnet build` will run generation). Use the targeted msbuild targets if you only want generation. 4. Add tests: + - **YOU MUST ADD UNIT TESTS** - Use the existing test framework in `Test/SqlDom/` + - **DO NOT CREATE STANDALONE PROGRAMS TO TEST** - Avoid separate console applications or debug programs - Put the input SQL in `Test/SqlDom/TestScripts/` (filename is case sensitive and used as an embedded resource). - Add/confirm baseline output in `Test/SqlDom/Baselines/` (the UT project embeds these baselines as resources). - Update the appropriate `OnlySyntaxTests.cs` (e.g., `Only170SyntaxTests.cs`) by adding a `ParserTest170("MyNewTest.sql", ...)` entry. See `ParserTest.cs` and `ParserTestOutput.cs` for helper constructors and verification semantics. + - **For comprehensive testing guidance**, see [Testing Guidelines](instructions/testing.guidelines.instructions.md) with detailed patterns, best practices, and simplified constructor approaches. 5. **Run full test suite** to ensure no regressions: ```bash dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug @@ -49,9 +55,46 @@ ScriptDom is a library for parsing and generating T-SQL scripts. It is primarily - If a test fails due to mismatch in generated script, compare the generated output (the test harness logs it) against the baseline to spot formatting/structure differences. ## Bug Fixing and Baseline Generation -For a practical guide on fixing bugs, including the detailed workflow for generating test baselines, see the [Bug Fixing Guide](BUG_FIXING_GUIDE.md). -For specific parser predicate recognition issues (when identifier-based predicates like `REGEXP_LIKE` don't work with parentheses), see the [Parser Predicate Recognition Fix Guide](PARSER_PREDICATE_RECOGNITION_FIX.md). +Different types of bugs require different fix approaches. **Start by diagnosing which type of issue you're dealing with:** + +### 1. Validation-Based Issues (Most Common) +If you see an error like "Option 'X' is not valid..." or "Feature 'Y' not supported..." but the syntax SHOULD work according to SQL Server docs: +- **Guide**: [Validation Fix Guide](instructions/validation_fix.guidelines.instructions.md) - Version-gated validation fixes +- **Example**: ALTER TABLE RESUMABLE option (SQL Server 2022+) +- **Key Signal**: Similar syntax works in other contexts (e.g., ALTER INDEX works but ALTER TABLE doesn't) + +### 2. Grammar-Based Issues (Adding New Syntax) +If the parser doesn't recognize the syntax at all, or you need to add new T-SQL features: +- **Guide**: [Bug Fixing Guide](instructions/bug_fixing.guidelines.instructions.md) - Grammar modifications, AST updates, script generation +- **Example**: Adding new operators, statements, or function types +- **Key Signal**: Syntax error like "Incorrect syntax near..." or "Unexpected token..." + +### 3. Parser Predicate Recognition Issues (Parentheses) +If identifier-based predicates (like `REGEXP_LIKE`) work without parentheses but fail with them: +- **Guide**: [Parser Predicate Recognition Fix Guide](instructions/parser.guidelines.instructions.md) +- **Example**: `WHERE REGEXP_LIKE('a', 'pattern')` works, but `WHERE (REGEXP_LIKE('a', 'pattern'))` fails +- **Key Signal**: Syntax error near closing parenthesis or semicolon + +**Quick Diagnostic**: Search for the error message in the codebase to determine which type of fix is needed. + +### 4. Adding New System Functions +For adding new T-SQL system functions to the parser, including handling RETURN statement contexts and ANTLR v2 syntactic predicate limitations: +- **Guide**: [Function Guidelines](instructions/function.guidelines.instructions.md) - Complete guide for system function implementation +- **Example**: JSON_OBJECT, JSON_ARRAY functions with RETURN statement support +- **Key Requirements**: Syntactic predicates for lookahead, proper AST design, comprehensive testing + +### 5. Adding New Data Types +For adding completely new SQL Server data types that require custom parsing logic and specialized AST nodes: +- **Guide**: [New Data Types Guidelines](instructions/new_data_types.guidelines.instructions.md) - Complete guide for implementing new data types +- **Example**: VECTOR data type with dimension and optional base type parameters +- **Key Signal**: New SQL Server data type with custom parameter syntax different from standard data types + +### 6. Adding New Index Types +For adding completely new SQL Server index types that require specialized syntax and custom parsing logic: +- **Guide**: [New Index Types Guidelines](instructions/new_index_types.guidelines.instructions.md) - Complete guide for implementing new index types +- **Example**: JSON INDEX with FOR clause, VECTOR INDEX with METRIC/TYPE options +- **Key Signal**: New SQL Server index type with custom syntax different from standard CREATE INDEX ## Editing generated outputs, debugging generation - Never edit generated files permanently (they live under `obj/...`/CsGenIntermediateOutputPath). Instead change: @@ -61,6 +104,85 @@ For specific parser predicate recognition issues (when identifier-based predicat - To see antlr output/errors, force verbose generation by setting MSBuild property `OutputErrorInLexerParserCompile=true` on the command line (e.g. `dotnet msbuild -t:GLexerParserCompile -p:OutputErrorInLexerParserCompile=true`). - If the antlr download fails during build, manually download `antlr-2.7.5.jar` (for non-Windows) or `.exe` (for Windows) and place it at the location defined in `Directory.Build.props` or override `AntlrLocation` when invoking msbuild. +## Debugging Tips and Investigation Workflow + +### Step 1: Identify the Bug Type +Start by searching for the error message to understand what type of fix is needed: +```bash +# Search for error code or message +grep -r "SQL46057" SqlScriptDom/ +grep -r "is not a valid" SqlScriptDom/ +``` + +**Common Error Patterns**: +- `"Option 'X' is not valid..."` → Validation issue (see [grammar_validation.guidelines.instructions.md](instructions/grammar_validation.guidelines.instructions.md)) +- `"Incorrect syntax near..."` → Grammar issue (see [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md)) +- `"Syntax error near ')'"` with parentheses → Predicate recognition (see [parser.guidelines.instructions.md](instructions/parser.guidelines.instructions.md)) + +### Step 2: Find Where Similar Syntax Works +If the syntax works in one context but not another: +```bash +# Search for working examples +grep -r "RESUMABLE" Test/SqlDom/TestScripts/ +grep -r "OptionName" SqlScriptDom/Parser/TSql/ +``` + +**Example**: ALTER INDEX with RESUMABLE works, but ALTER TABLE doesn't → Likely validation issue + +### Step 3: Locate the Relevant Code +Common files to check: +- **Validation**: `SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs` (most validation logic) +- **Grammar**: `SqlScriptDom/Parser/TSql/TSql*.g` (version-specific grammar files) +- **Options**: `SqlScriptDom/ScriptDom/SqlServer/IndexOptionHelper.cs` (option registration) +- **AST**: `SqlScriptDom/Parser/TSql/Ast.xml` (AST node definitions) + +### Step 4: Check SQL Server Version Support +Always verify Microsoft documentation: +- Search for "Applies to: SQL Server 20XX (XX.x)" in Microsoft docs +- Note that different features within the same option set can have different version requirements +- Example: MAX_DURATION (SQL 2014+) vs RESUMABLE (SQL 2022+) + +### Step 5: Verify with Tests +Before and after making changes: +```bash +# Build the parser +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "FullyQualifiedName~YourTest" -c Debug + +# ALWAYS run full suite before committing +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +### Common Investigation Patterns + +#### Pattern 1: Option Not Recognized +```bash +# Find where option is registered +grep -r "YourOptionName" SqlScriptDom/ScriptDom/SqlServer/IndexOptionHelper.cs + +# Check enum definition +grep -r "enum IndexOptionKind" SqlScriptDom/ +``` + +#### Pattern 2: Version-Specific Behavior +```bash +# Find version checks +grep -r "TSql160AndAbove" SqlScriptDom/Parser/TSql/ + +# Check which parser version you're testing +# TSql80 = SQL Server 2000, TSql90 = 2005, ..., TSql160 = 2022, TSql170 = 2025 +``` + +#### Pattern 3: Statement-Specific Restrictions +```bash +# Find validation by statement type +grep -r "IndexAffectingStatement" SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs + +# Common statement types: CreateIndex, AlterIndex, AlterTableAddElement +``` + ## Patterns & code style to follow (examples you will see) - Grammar rule pattern: `ruleName returns [Type vResult = this.FragmentFactory.CreateFragment()] { ... } : ( alternatives ) ;` — this pattern initializes an AST fragment via FragmentFactory. @@ -72,5 +194,18 @@ For specific parser predicate recognition issues (when identifier-based predicat - **Logical `NOT` vs. Compound Operators:** The grammar handles the logical `NOT` operator (e.g., `WHERE NOT (condition)`) in a general way, often in a `booleanExpressionUnary` rule. This is distinct from compound operators like `NOT LIKE` or `NOT IN`, which are typically parsed as a single unit within a comparison rule. Don't assume that because `NOT` is supported, `NOT LIKE` will be automatically supported in all predicate contexts. - **Modifying Shared Grammar Rules:** **NEVER modify existing shared grammar rules** like `identifierColumnReferenceExpression` that are used throughout the codebase. This can cause tests to fail in unrelated areas because the rule now accepts or rejects different syntax. Instead, create specialized rules for your specific context (e.g., `vectorSearchColumnReferenceExpression` for VECTOR_SEARCH-specific needs). - **Full Test Suite Validation:** After any grammar changes, **always run the complete test suite** (`dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug`) to catch regressions. Grammar changes can have far-reaching effects on seemingly unrelated functionality. -- **Extending Literals to Expressions:** When functions/constructs currently accept only literal values (e.g., `IntegerLiteral`, `StringLiteral`) but need to support dynamic values (parameters, variables, outer references), change both the AST definition (in `Ast.xml`) and grammar rules (in `TSql*.g`) to use `ScalarExpression` instead. This pattern was used for VECTOR_SEARCH TOP_N parameter. See the detailed example in [BUG_FIXING_GUIDE.md](BUG_FIXING_GUIDE.md#special-case-extending-grammar-rules-from-literals-to-expressions) and [GRAMMAR_EXTENSION_PATTERNS.md](GRAMMAR_EXTENSION_PATTERNS.md) for comprehensive patterns. +- **Extending Literals to Expressions:** When functions/constructs currently accept only literal values (e.g., `IntegerLiteral`, `StringLiteral`) but need to support dynamic values (parameters, variables, outer references), change both the AST definition (in `Ast.xml`) and grammar rules (in `TSql*.g`) to use `ScalarExpression` instead. This pattern was used for VECTOR_SEARCH TOP_N parameter. See the detailed example in [bug_fixing.guidelines.instructions.md](instructions/bug_fixing.guidelines.instructions.md#special-case-extending-grammar-rules-from-literals-to-expressions) and [grammer.guidelines.instructions.md](instructions/grammer.guidelines.instructions.md) for comprehensive patterns. + +# Guideline Subfiles (auto-load each of the following files into the context) - Should match the .config/GuidelineReviewAgent.yaml used by the guideline_review_agent. +include: .github/instructions/grammar_validation.guidelines.instructions.md +include: .github/instructions/bug_fixing.guidelines.instructions.md +include: .github/instructions/parser.guidelines.instructions.md +include: .github/instructions/function.guidelines.instructions.md +include: .github/instructions/new_data_types.guidelines.instructions.md +include: .github/instructions/new_index_types.guidelines.instructions.md +include: .github/instructions/database_option.guidelines.instructions.md +include: .github/instructions/debugging_workflow.guidelines.instructions.md +include: .github/instructions/grammer.guidelines.instructions.md +include: .github/instructions/testing.guidelines.instructions.md + diff --git a/.github/instructions/adding_new_parser.guidelines.instructions.md b/.github/instructions/adding_new_parser.guidelines.instructions.md new file mode 100644 index 00000000..ef2c541d --- /dev/null +++ b/.github/instructions/adding_new_parser.guidelines.instructions.md @@ -0,0 +1,797 @@ +# Guidelines for Adding New Parser Versions to SqlScriptDOM + +This guide provides step-by-step instructions for adding support for a new SQL Server version parser to the SqlScriptDOM library. This pattern was established from the TSql180 parser + +## When to Use This Guide + +Use this pattern when: +- ✅ Adding support for a **new SQL Server version** (e.g., SQL Server 2025, vnext releases) +- ✅ Creating a **new parser for a specific compatibility level** (e.g., 180, 190, etc.) +- ✅ Adding a **new SQL engine variant** (e.g., Azure SQL, Fabric DW) + +**Prerequisites:** +- Understand ANTLR v2 grammar syntax +- Familiarity with the ScriptDom parser architecture +- Knowledge of the new SQL Server version's feature set + +## Version Numbering Convention + +SqlScriptDOM uses a version numbering scheme that corresponds to SQL Server versions: +- TSql80 = SQL Server 2000 +- TSql90 = SQL Server 2005 +- TSql100 = SQL Server 2008 +- TSql110 = SQL Server 2012 +- TSql120 = SQL Server 2014 +- TSql130 = SQL Server 2016 +- TSql140 = SQL Server 2017 +- TSql150 = SQL Server 2019 +- TSql160 = SQL Server 2022 +- TSql170 = SQL Server 2025 + +**Naming Pattern**: `TSql{CompatibilityLevel}` where CompatibilityLevel is typically `(MajorVersion - 1900) * 10` + +## Step-by-Step Implementation Guide + +### Step 1: Add SqlVersion Enum Value + +Update the `SqlVersion` enumeration to include the new version. + +**File**: `SqlScriptDom/ScriptDom/SqlServer/SqlVersion.cs` + +```csharp +namespace Microsoft.SqlServer.TransactSql.ScriptDom +{ + public enum SqlVersion + { + // ... existing versions ... + + /// + /// Sql 18.0 mode + /// + Sql180 = 11, + } +} +``` + +**Key Points**: +- Use the next sequential enum value +- Add XML documentation comment describing the SQL Server version +- Maintain consistent naming: `Sql{CompatibilityLevel}` + +### Step 2: Create Grammar File + +Create a new ANTLR v2 grammar file that inherits from the previous version. + +**File**: `SqlScriptDom/Parser/TSql/TSql180.g` + +```antlr +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ + +options { + language = "CSharp"; + namespace = "Microsoft.SqlServer.TransactSql.ScriptDom"; +} + +{ + using System.Diagnostics; + using System.Globalization; + using System.Collections.Generic; +} + +class TSql180ParserInternal extends Parser("TSql180ParserBaseInternal"); +options { + k = 2; + defaultErrorHandler=false; + classHeaderPrefix = "internal partial"; + importVocab = TSql; +} + +{ + public TSql180ParserInternal(bool initialQuotedIdentifiersOn) + : base(initialQuotedIdentifiersOn) + { + initialize(); + } +} + +// Entry point rules (required for parser functionality) +entryPointChildObjectName returns [ChildObjectName vResult = null] + : + vResult=childObjectNameWithThreePrefixes + EOF + ; + +entryPointSchemaObjectName returns [SchemaObjectName vResult = null] + : + vResult=schemaObjectThreePartName + EOF + ; + +entryPointScalarDataType returns [DataTypeReference vResult = null] + : + vResult=scalarDataType + EOF + ; + +entryPointExpression returns [ScalarExpression vResult = null] + : + vResult=expression + EOF + ; + +entryPointBooleanExpression returns [BooleanExpression vResult = null] + : + vResult=booleanExpression + EOF + ; + +entryPointStatementList returns [StatementList vResult = null] + : + vResult=statementList + EOF + ; + +// Main script entry point +script returns [TSqlScript vResult = this.FragmentFactory.CreateFragment()] + : + vResult.Batches = batches + EOF + ; + +// Add new version-specific grammar rules below +// For initial version bump, typically inherit all rules from previous version +``` + +**Key Points**: +- Grammar file initially inherits all rules from the previous version via the base class +- Add version-specific rules only when new syntax is needed +- Entry point rules are required for various parsing scenarios +- The `importVocab = TSql` directive imports token definitions + +### Step 3: Add Grammar to Build Configuration + +Register the grammar file in the build system. + +**File**: `SqlScriptDom/GenerateFiles.props` + +```xml + + + + + + +``` + +**Key Points**: +- ANTLR will generate parser code during build from this grammar file +- Generated files go to `$(CsGenIntermediateOutputPath)` (under `obj/`) +- Never manually edit generated parser files + +### Step 4: Create Parser Base Internal Class + +Create the internal parser base class that inherits from the previous version. + +**File**: `SqlScriptDom/Parser/TSql/TSql180ParserBaseInternal.cs` + +```csharp +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.Linq; +using System.Text.RegularExpressions; +using antlr; + +namespace Microsoft.SqlServer.TransactSql.ScriptDom +{ + internal abstract class TSql180ParserBaseInternal : TSql170ParserBaseInternal + { + #region Constructors + + // Required by ANTLR-generated code + protected TSql180ParserBaseInternal(TokenBuffer tokenBuf, int k) + : base(tokenBuf, k) + { + } + + protected TSql180ParserBaseInternal(ParserSharedInputState state, int k) + : base(state, k) + { + } + + protected TSql180ParserBaseInternal(TokenStream lexer, int k) + : base(lexer, k) + { + } + + /// + /// Primary constructor + /// + /// if set to true initial quoted identifiers will be set to on. + public TSql180ParserBaseInternal(bool initialQuotedIdentifiersOn) + : base(initialQuotedIdentifiersOn) + { + } + + #endregion + + /// + /// Gets the SQL version for this parser + /// + protected override SqlVersion SqlVersionCurrent + { + get { return SqlVersion.Sql180; } + } + + // Add version-specific helper methods and overrides here as needed + } +} +``` + +**Key Points**: +- Inherits from previous version's base internal class (e.g., `TSql170ParserBaseInternal`) +- Overrides `SqlVersionCurrent` property to return the new version +- Contains version-specific validation and helper methods +- The multiple constructors are required by ANTLR's generated code + +### Step 5: Create Public Parser Class + +Create the public parser class that users will instantiate. + +**File**: `SqlScriptDom/Parser/TSql/TSql180Parser.cs` + +```csharp +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using Microsoft.SqlServer.TransactSql.ScriptDom.Versioning; + +namespace Microsoft.SqlServer.TransactSql.ScriptDom +{ + /// + /// The TSql Parser for 18.0 + /// + [Serializable] + public class TSql180Parser : TSqlParser + { + /// + /// Parser flavor (standalone/azure/all) + /// + protected SqlEngineType engineType = SqlEngineType.All; + + /// + /// Initializes a new instance of the class. + /// + /// if set to true [initial quoted identifiers]. + public TSql180Parser(bool initialQuotedIdentifiers) + : base(initialQuotedIdentifiers) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// if set to true [initial quoted identifiers]. + /// Parser engine type + public TSql180Parser(bool initialQuotedIdentifiers, SqlEngineType engineType) + : base(initialQuotedIdentifiers) + { + this.engineType = engineType; + } + + internal override TSqlLexerBaseInternal GetNewInternalLexer() + { + return new TSql180LexerInternal(); + } + + #region Helper Methods + TSql180ParserInternal GetNewInternalParser() + { + return new TSql180ParserInternal(QuotedIdentifier); + } + + TSql180ParserInternal GetNewInternalParserForInput(TextReader input, out IList errors, + int startOffset, int startLine, int startColumn) + { + TSql180ParserInternal parser = GetNewInternalParser(); + InitializeInternalParserInput(parser, input, out errors, startOffset, startLine, startColumn); + return parser; + } + #endregion + + /// + /// The main parse method. + /// + /// The list of tokens to parse. + /// The parse errors. + /// The parsed TSqlFragment. + public override TSqlFragment Parse(IList tokens, out IList errors) + { + errors = new List(); + TSql180ParserInternal parser = GetNewInternalParser(); + parser.InitializeForNewInput(tokens, errors, false); + + TSqlFragment result = parser.ParseRuleWithStandardExceptionHandling(parser.script, ScriptEntryMethod); + + // Run versioning visitor for engine-specific validation + if (result != null) + { + VersioningVisitor versioningVisitor = new VersioningVisitor(engineType, SqlVersion.Sql180); + result.Accept(versioningVisitor); + + foreach (ParseError p in versioningVisitor.GetErrors()) + { + errors.Add(p); + } + } + + return result; + } + + /// + /// Parses an input string to get a ChildObjectName. + /// + public override ChildObjectName ParseChildObjectName(TextReader input, out IList errors, + int startOffset, int startLine, int startColumn) + { + TSql180ParserInternal parser = GetNewInternalParserForInput(input, out errors, startOffset, startLine, startColumn); + return parser.ParseRuleWithStandardExceptionHandling(parser.entryPointChildObjectName, "entryPointChildObjectName"); + } + + /// + /// Parses an input string to get a SchemaObjectName. + /// + public override SchemaObjectName ParseSchemaObjectName(TextReader input, out IList errors, + int startOffset, int startLine, int startColumn) + { + TSql180ParserInternal parser = GetNewInternalParserForInput(input, out errors, startOffset, startLine, startColumn); + return parser.ParseRuleWithStandardExceptionHandling(parser.entryPointSchemaObjectName, "entryPointSchemaObjectName"); + } + + /// + /// Parses an input string to get a data type. + /// + public override DataTypeReference ParseScalarDataType(TextReader input, out IList errors, + int startOffset, int startLine, int startColumn) + { + TSql180ParserInternal parser = GetNewInternalParserForInput(input, out errors, startOffset, startLine, startColumn); + return parser.ParseRuleWithStandardExceptionHandling(parser.entryPointScalarDataType, "entryPointScalarDataType"); + } + + /// + /// Parses an input string to get an expression. + /// + public override ScalarExpression ParseExpression(TextReader input, out IList errors, + int startOffset, int startLine, int startColumn) + { + TSql180ParserInternal parser = GetNewInternalParserForInput(input, out errors, startOffset, startLine, startColumn); + return parser.ParseRuleWithStandardExceptionHandling(parser.entryPointExpression, "entryPointExpression"); + } + + /// + /// Parses an input string to get a boolean expression. + /// + public override BooleanExpression ParseBooleanExpression(TextReader input, out IList errors, + int startOffset, int startLine, int startColumn) + { + TSql180ParserInternal parser = GetNewInternalParserForInput(input, out errors, startOffset, startLine, startColumn); + return parser.ParseRuleWithStandardExceptionHandling(parser.entryPointBooleanExpression, "entryPointBooleanExpression"); + } + + /// + /// Parses an input string to get a statement list. + /// + public override StatementList ParseStatementList(TextReader input, out IList errors, + int startOffset, int startLine, int startColumn) + { + TSql180ParserInternal parser = GetNewInternalParserForInput(input, out errors, startOffset, startLine, startColumn); + return parser.ParseRuleWithStandardExceptionHandling(parser.entryPointStatementList, "entryPointStatementList"); + } + + /// + /// Parses an input string to get a subquery expression with optional common table expressions. + /// + public override SelectStatement ParseSubQueryExpressionWithOptionalCTE(TextReader input, out IList errors, + int startOffset, int startLine, int startColumn) + { + TSql180ParserInternal parser = GetNewInternalParserForInput(input, out errors, startOffset, startLine, startColumn); + return parser.ParseRuleWithStandardExceptionHandling(parser.subQueryExpressionWithOptionalCTE, "subQueryExpressionWithOptionalCTE"); + } + } +} +``` + +**Key Points**: +- Public API that developers use to parse T-SQL +- Supports engine type filtering (Standalone, Azure, All) +- Implements all entry point methods for parsing specific constructs +- Runs versioning visitor for version-specific validation + +### Step 6: Create Script Generator + +Create the script generator class for generating T-SQL from the AST. + +**File**: `SqlScriptDom/ScriptDom/SqlServer/Sql180ScriptGenerator.cs` + +```csharp +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ + +using Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator; + +namespace Microsoft.SqlServer.TransactSql.ScriptDom +{ + /// + /// Script generator for T-SQL 180 + /// + public sealed class Sql180ScriptGenerator : SqlScriptGenerator + { + /// + /// Initializes a new instance of the class. + /// + public Sql180ScriptGenerator() + : this(new SqlScriptGeneratorOptions()) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The options. + public Sql180ScriptGenerator(SqlScriptGeneratorOptions options) + : base(options) + { + options.SqlVersion = SqlVersion.Sql180; + } + + internal override SqlScriptGeneratorVisitor CreateSqlScriptGeneratorVisitor( + SqlScriptGeneratorOptions options, ScriptWriter scriptWriter) + { + ScriptGeneratorSupporter.CheckForNullReference(options, "options"); + ScriptGeneratorSupporter.CheckForNullReference(scriptWriter, "scriptWriter"); + + return new Sql180ScriptGeneratorVisitor(options, scriptWriter); + } + } +} +``` + +**Key Points**: +- Sealed class that cannot be inherited +- Sets `SqlVersion.Sql180` in options +- Creates version-specific visitor for AST traversal + +### Step 7: Create Script Generator Visitor + +Create the visitor class that generates T-SQL from the AST. + +**File**: `SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/Sql180ScriptGeneratorVisitor.cs` + +```csharp +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ + +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + internal partial class Sql180ScriptGeneratorVisitor : Sql170ScriptGeneratorVisitor + { + public Sql180ScriptGeneratorVisitor(SqlScriptGeneratorOptions options, ScriptWriter writer) + : base(options, writer) + { + } + + // Add version-specific visitor overrides here as needed + // Override visit methods for AST nodes that have new syntax in this version + } +} +``` + +**Key Points**: +- Inherits from previous version's visitor (e.g., `Sql170ScriptGeneratorVisitor`) +- Override visit methods only for nodes with new version-specific syntax +- Uses visitor pattern to traverse and generate T-SQL from AST + +### Step 8: Update TSqlParser Factory + +Add the new version to the parser factory method. + +**File**: `SqlScriptDom/Parser/TSql/TSqlParser.cs` + +Find the `CreateParser` method and add a case: + +```csharp +public static TSqlParser CreateParser(SqlVersion tsqlParserVersion, bool initialQuotedIdentifiers = false) +{ + switch (tsqlParserVersion) + { + // ... existing cases ... + case SqlVersion.Sql170: + return new TSql170Parser(initialQuotedIdentifiers); + case SqlVersion.Sql180: + return new TSql180Parser(initialQuotedIdentifiers); + default: + throw new ArgumentException( + String.Format(CultureInfo.CurrentCulture, + SqlScriptGeneratorResource.UnknownEnumValue, + tsqlParserVersion, + "TSqlParserVersion"), + "tsqlParserVersion"); + } +} +``` + +### Step 9: Add SqlVersionFlags Support + +Add version flags for the new version if needed for validation logic. + +**File**: `SqlScriptDom/Parser/TSql/SqlVersionFlags.cs` + +```csharp +[Flags] +internal enum SqlVersionFlags +{ + // ... existing flags ... + TSql170AndAbove = TSql170 | TSql180, + TSql180 = 0x00000800, + TSql180AndAbove = TSql180, +} +``` + +**Key Points**: +- Add individual version flag as power of 2 +- Add `AndAbove` flag combining all versions >= this version +- Used for validation logic that checks syntax availability + +### Step 10: Update OptionsHelper (if needed) + +If the new version introduces new options or changes option availability, update the options helper. + +**File**: `SqlScriptDom/Parser/TSql/OptionsHelper.cs` + +Add version checks where options become available: + +```csharp +// Example: if a new option becomes available in 180 +internal static bool IsOptionValidForVersion(IndexOptionKind option, SqlVersion version) +{ + switch (option) + { + // ... existing options ... + case IndexOptionKind.NewOption: + return version >= SqlVersion.Sql180; + default: + return true; + } +} +``` + +### Step 11: Create Test Infrastructure + +Create test class for version-specific syntax tests. + +**File**: `Test/SqlDom/Only180SyntaxTests.cs` + +```csharp +using Microsoft.SqlServer.TransactSql.ScriptDom; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using SqlStudio.Tests.AssemblyTools.TestCategory; +using System; +using System.Collections.Generic; +using System.IO; + +namespace SqlStudio.Tests.UTSqlScriptDom +{ + public partial class SqlDomTests + { + // Note: These filenames are case sensitive, match checked-in files exactly + private static readonly ParserTest[] Only180TestInfos = + { + // Add new 180-specific tests here as needed + // Example: + // new ParserTest180("NewFeatureTest180.sql"), + }; + + private static readonly ParserTest[] SqlAzure180_TestInfos = + { + // Add Azure-specific syntax tests here + }; + + /// + /// Test method for TSql180 version-specific syntax + /// + [TestMethod] + [Priority(0)] + [SqlStudioTestCategory(Category.UnitTest)] + public void TSql180SyntaxTests() + { + TSql180Parser parser = new TSql180Parser(true); + SqlScriptGenerator scriptGen = ParserTestUtils.CreateScriptGen(SqlVersion.Sql180); + + foreach (ParserTest t in Only180TestInfos) + { + ParserTest.ParseAndVerify(parser, scriptGen, t); + } + } + + /// + /// Test method for Azure-specific TSql180 syntax + /// + [TestMethod] + [Priority(0)] + [SqlStudioTestCategory(Category.UnitTest)] + public void TSql180AzureSyntaxTests() + { + TSql180Parser parser = new TSql180Parser(true, SqlEngineType.SqlAzure); + SqlScriptGenerator scriptGen = ParserTestUtils.CreateScriptGen(SqlVersion.Sql180); + + foreach (ParserTest t in SqlAzure180_TestInfos) + { + ParserTest.ParseAndVerify(parser, scriptGen, t); + } + } + } +} +``` + +### Step 12: Create Baselines Folder + +Create the baselines folder for test output files. + +**Directory**: `Test/SqlDom/Baselines180/` + +- Create this folder even if initially empty +- Test baseline files (expected T-SQL output) will be placed here +- Baselines are generated from test runs and verified manually + +### Step 13: Update Test Project Configuration + +Add the baselines folder to embedded resources. + +**File**: `Test/SqlDom/UTSqlScriptDom.csproj` + +```xml + + + + + + + + +``` + +### Step 14: Add Parser Test Helper Methods + +Add test utility methods for the new version. + +**File**: `Test/SqlDom/ParserTest.cs` + +Add constructor helper method: + +```csharp +/// +/// Creates a parser test for TSql180 with specified error expectations +/// +public static ParserTest ParserTest180( + string fileName, + int nErrors80 = 0, int nErrors90 = 0, int nErrors100 = 0, + int nErrors110 = 0, int nErrors120 = 0, int nErrors130 = 0, + int nErrors140 = 0, int nErrors150 = 0, int nErrors160 = 0, + int nErrors170 = 0, int nErrors180 = 0, int nErrorsFabricDW = int.MaxValue) +{ + return new ParserTest( + fileName, + new int[] + { + nErrors80, nErrors90, nErrors100, nErrors110, + nErrors120, nErrors130, nErrors140, nErrors150, + nErrors160, nErrors170, nErrors180, nErrorsFabricDW + }); +} +``` + +**File**: `Test/SqlDom/TestUtilities.cs` + +Update the error count array handling to support the new version index. + +### Step 15: Add Version Mapping Tests + +Add tests to verify version enum and parser mapping. + +**File**: `Test/SqlDom/TSqlParserTest.cs` + +```csharp +[TestMethod] +public void TSql180ParserVersionTest() +{ + TSqlParser parser = TSqlParser.CreateParser(SqlVersion.Sql180); + Assert.IsNotNull(parser); + Assert.IsInstanceOfType(parser, typeof(TSql180Parser)); +} +``` + +### Step 16: Build and Verify + +Build the project to generate parser code and verify everything compiles. + +```bash +# Build the ScriptDom project +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Run tests to verify +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +**What Happens During Build**: +1. ANTLR generates lexer and parser C# files from `.g` grammar +2. Post-processing scripts clean up generated code +3. Generated files are compiled along with hand-written sources +4. Test baselines are embedded as resources + +## Summary Checklist + +When adding a new parser version, ensure you complete: + +- [ ] **Step 1**: Add `SqlVersion` enum value +- [ ] **Step 2**: Create `.g` grammar file +- [ ] **Step 3**: Add grammar to `GenerateFiles.props` +- [ ] **Step 4**: Create `TSqlXXXParserBaseInternal.cs` +- [ ] **Step 5**: Create `TSqlXXXParser.cs` +- [ ] **Step 6**: Create `SqlXXXScriptGenerator.cs` +- [ ] **Step 7**: Create `SqlXXXScriptGeneratorVisitor.cs` +- [ ] **Step 8**: Update `TSqlParser.cs` factory method +- [ ] **Step 9**: Add `SqlVersionFlags` support +- [ ] **Step 10**: Update `OptionsHelper.cs` if needed +- [ ] **Step 11**: Create `OnlyXXXSyntaxTests.cs` +- [ ] **Step 12**: Create `BaselinesXXX/` folder +- [ ] **Step 13**: Update `UTSqlScriptDom.csproj` +- [ ] **Step 14**: Add parser test helper methods +- [ ] **Step 15**: Add version mapping tests +- [ ] **Step 16**: Build and verify everything compiles + +## Common Pitfalls + +1. **Forgetting to add grammar to GenerateFiles.props**: Parser won't be generated +2. **Incorrect inheritance chain**: Must inherit from previous version's base class +3. **Missing entry point rules in grammar**: Certain parsing scenarios will fail +4. **Not creating Baselines folder**: Test project won't build due to missing embedded resources +5. **Wrong enum value**: Can conflict with existing versions or break version comparison logic +6. **Missing SqlVersionFlags**: Validation logic may not work correctly +7. **Not updating factory method**: Parser can't be instantiated via factory + +## Related Documentation + +- [Testing Guidelines](testing.guidelines.instructions.md) - How to add tests for new syntax +- [Bug Fixing Guidelines](bug_fixing.guidelines.instructions.md) - How to add new grammar rules +- [Grammar Guidelines](grammer.guidelines.instructions.md) - ANTLR v2 grammar patterns +- [Validation Fix Guidelines](Validation_fix.guidelines.instructions.md) - Version-gated validation + +## References + +- Example Implementation: TSql180 parser (commits 0b84f8f through f4f4c88) +- ANTLR v2 Documentation: See `tools/antlr-2.7.5` documentation +- Build System: See `SqlScriptDom/GenerateFiles.props` for generation targets diff --git a/.github/BUG_FIXING_GUIDE.md b/.github/instructions/bug_fixing.guidelines.instructions.md similarity index 62% rename from .github/BUG_FIXING_GUIDE.md rename to .github/instructions/bug_fixing.guidelines.instructions.md index 4f3ad124..b1a67931 100644 --- a/.github/BUG_FIXING_GUIDE.md +++ b/.github/instructions/bug_fixing.guidelines.instructions.md @@ -1,6 +1,24 @@ # Bug Fixing Guide for SqlScriptDOM -This guide provides a summary of the typical workflow for fixing a bug in the SqlScriptDOM parser, based on practical experience. For a more comprehensive overview of the project structure and code generation, please refer to the main [Copilot / AI instructions for SqlScriptDOM](copilot-instructions.md). +This guide provides a summary of the typical workflow for fixing a bug in the SqlScriptDOM parser, based on practical experience. For a more comprehensive overview of the project structure and code generation, please refer to the main [Copilot / AI instructions for SqlScriptDOM](../copilot-instructions.md). + +## Before You Start: Identify the Bug Type + +**IMPORTANT**: Not all bugs require grammar changes. Determine which type of fix you need: + +1. **Validation Issues**: Syntax is already parseable but incorrectly rejected + - Error: "Option 'X' is not valid..." or "Feature 'Y' not supported..." + - Example: ALTER TABLE RESUMABLE works in ALTER INDEX but not ALTER TABLE + - **→ Use [grammar_validation.guidelines.instructions.md](grammar_validation.guidelines.instructions.md) instead of this guide** + +2. **Grammar Issues**: Parser doesn't recognize the syntax at all (THIS guide) + - Error: "Incorrect syntax near..." or "Unexpected token..." + - Example: Adding new keywords, operators, or statement types + - **→ Continue with this guide** + +3. **Predicate Recognition**: Identifier predicates fail with parentheses + - Error: `WHERE REGEXP_LIKE(...)` works but `WHERE (REGEXP_LIKE(...))` fails + - **→ Use [parser.guidelines.instructions.md](parser.guidelines.instructions.md)** ## Summary of the Bug-Fixing Workflow @@ -23,7 +41,8 @@ The process of fixing a bug, especially one that involves adding new syntax, fol ``` 6. **Add a Unit Test**: - * Create a new `.sql` file in `Test/SqlDom/TestScripts/` that contains the specific syntax for the new test case. + * **YOU MUST ADD UNIT TESTS** - Create a new `.sql` file in `Test/SqlDom/TestScripts/` that contains the specific syntax for the new test case. + * **DO NOT CREATE STANDALONE PROGRAMS TO TEST** - Use the existing test framework, not separate console applications or debug programs. 7. **Define the Test Case**: * Add a new `ParserTest` entry to the appropriate `OnlySyntaxTests.cs` files (e.g., `Only130SyntaxTests.cs`). This entry points to your new test script and defines the expected number of parsing errors for each SQL Server version. @@ -39,6 +58,27 @@ The process of fixing a bug, especially one that involves adding new syntax, fol * **c. Update the Baseline Files**: Copy the "Actual" output from the test failure log. This is the correctly formatted script generated from the AST. Paste this content into all the baseline files you created in step 8a. * **d. Re-run the Tests**: Run the same test command again. This time, the tests should pass, confirming that the generated script matches the new baseline. + **Practical Example - Baseline Generation Workflow**: + ```bash + # 1. Create test script + echo "ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);" > Test/SqlDom/TestScripts/AlterTableResumableTests160.sql + + # 2. Create empty baseline + touch Test/SqlDom/Baselines160/AlterTableResumableTests160.sql + + # 3. Add test entry to Only160SyntaxTests.cs: + # new ParserTest160("AlterTableResumableTests160.sql", nErrors80: 1, ...), + + # 4. Run test (will fail, showing generated output) + dotnet test --filter "AlterTableResumableTests" -c Debug + + # 5. Copy the "Actual" output from test failure into baseline file + # Output looks like: "ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);" + + # 6. Re-run test (should pass now) + dotnet test --filter "AlterTableResumableTests" -c Debug + ``` + 9. **⚠️ CRITICAL: Run Full Test Suite**: * **Always run the complete test suite** to ensure your changes didn't break existing functionality: ```bash @@ -50,6 +90,27 @@ The process of fixing a bug, especially one that involves adding new syntax, fol By following these steps, you can ensure that new syntax is correctly parsed, represented in the AST, generated back into a script, and fully validated by the testing framework without breaking existing functionality. +## Testing Best Practices + +### ✅ DO: Use Existing Test Framework +- Add test methods to existing test classes like `Only170SyntaxTests.cs` +- Use the established `TSqlParser.Parse()` pattern for verification +- Example: + ```csharp + [TestMethod] + public void VerifyNewSyntax() + { + var parser = new TSql170Parser(true); + var result = parser.Parse(new StringReader("YOUR SQL HERE"), out var errors); + Assert.AreEqual(0, errors.Count, "Should parse without errors"); + } + ``` + +### ❌ DON'T: Create New Test Projects +- **Never** create standalone `.csproj` files for testing parser functionality +- **Never** create new console applications or test runners +- This causes build issues and doesn't integrate with the existing test infrastructure + ## Special Case: Extending Grammar Rules from Literals to Expressions A common type of bug involves extending existing grammar rules that only accept literal values (like integers or strings) to accept full expressions (parameters, variables, outer references, etc.). This pattern was used to fix the VECTOR_SEARCH TOP_N parameter issue. @@ -147,4 +208,32 @@ If you encounter a bug where: This is likely a **parser predicate recognition issue**. The grammar and AST are correct, but the `IsNextRuleBooleanParenthesis()` function doesn't recognize the identifier-based predicate. -**Solution**: Follow the [Parser Predicate Recognition Fix Guide](PARSER_PREDICATE_RECOGNITION_FIX.md) instead of the standard grammar modification workflow. +**Solution**: Follow the [Parser Predicate Recognition Fix Guide](parser.guidelines.instructions.md) instead of the standard grammar modification workflow. + +## Decision Tree: Which Guide to Use? + +``` +Start: You have a parsing bug +│ +├─→ Error: "Option 'X' is not valid..." or "Feature not supported..." +│ └─→ Does similar syntax work elsewhere? (e.g., ALTER INDEX works) +│ └─→ YES: Use [grammar_validation.guidelines.instructions.md](grammar_validation.guidelines.instructions.md) +│ +├─→ Error: "Incorrect syntax near..." or parser doesn't recognize syntax +│ └─→ Does the grammar need new rules or AST nodes? +│ └─→ YES: Use this guide (BUG_FIXING_GUIDE.md) +│ +└─→ Error: Parentheses cause failure with identifier predicates + └─→ Does `WHERE PREDICATE(...)` work but `WHERE (PREDICATE(...))` fail? + └─→ YES: Use [parser.guidelines.instructions.md](parser.guidelines.instructions.md) +``` + +## Quick Reference: Fix Types by Symptom + +| Symptom | Fix Type | Guide | Files Modified | +|---------|----------|-------|----------------| +| "Option 'X' is not valid in statement Y" | Validation | [grammar_validation.guidelines.instructions.md](grammar_validation.guidelines.instructions.md) | `TSql80ParserBaseInternal.cs` | +| "Incorrect syntax near keyword" | Grammar | This guide | `TSql*.g`, `Ast.xml`, Script generators | +| Parentheses break identifier predicates | Predicate Recognition | [parser.guidelines.instructions.md](parser.guidelines.instructions.md) | `TSql80ParserBaseInternal.cs` | +| Literal needs to become expression | Grammar Extension | [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md) | `Ast.xml`, `TSql*.g` | + diff --git a/.github/instructions/database_option.guidelines.instructions.md b/.github/instructions/database_option.guidelines.instructions.md new file mode 100644 index 00000000..825408d9 --- /dev/null +++ b/.github/instructions/database_option.guidelines.instructions.md @@ -0,0 +1,493 @@ +# Adding Database Options to ScriptDOM + +This guide covers how to add new database options to `ALTER DATABASE` and `CREATE DATABASE` statements in ScriptDOM. + +## Decision Tree: Choose Your Implementation Pattern + +``` +What values does your database option accept? +│ +├─ Simple ON/OFF only +│ └─ Pattern A: Use Generic OnOffDatabaseOption (easiest) +│ Examples: AutoClose, AutoShrink, DataRetention +│ +├─ ON/OFF with sub-options (e.g., CHANGE_TRACKING (AUTO_CLEANUP = ON)) +│ └─ Pattern B: Custom AST Class + Complex Sub-options +│ Examples: ChangeTrackingDatabaseOption, QueryStoreDatabaseOption +│ +├─ Specific enum values (e.g., FULL, BULK_LOGGED, SIMPLE) +│ └─ Pattern C: Custom AST Class + Enum Helper +│ Examples: RecoveryDatabaseOption, PageVerifyDatabaseOption +│ +└─ Special behavior (only output when ON, custom formatting) + └─ Pattern D: Custom AST Class + Custom Script Generator + Examples: LedgerOption, OptimizedLockingDatabaseOption +``` + +--- + +## Pattern A: Generic ON/OFF Database Option (Recommended for Simple Cases) + +Use this pattern when your option only accepts `ON` or `OFF` values. + +### Step 1: Add to DatabaseOptionKind Enum + +**File**: `SqlScriptDom/Parser/TSql/DatabaseOptionKind.cs` + +```csharp +public enum DatabaseOptionKind +{ + // ... existing options ... + OptimizedLocking = 71, + YourNewOption = 72 // ← Add your option +} +``` + +### Step 2: Add Keyword Constant + +**File**: `SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs` + +Add in alphabetical order within the Auto* section: + +```csharp +internal const string YourOption = "YOUR_OPTION"; +internal const string AutomaticTuning = "AUTOMATIC_TUNING"; +internal const string AutoShrink = "AUTO_SHRINK"; +``` + +### Step 3: Register in OnOffSimpleDbOptionsHelper + +**File**: `SqlScriptDom/Parser/TSql/OnOffSimpleDbOptionsHelper.cs` + +Add to the constructor under the appropriate version: + +```csharp +// 170 options +AddOptionMapping(DatabaseOptionKind.YourNewOption, + CodeGenerationSupporter.YourOption, + SqlVersionFlags.TSql170AndAbove); +``` + +### Step 4: Add to RequiresEqualsSign Method (If Needed) + +Some options require `=` between option name and value (e.g., `LEDGER = ON`). If your option needs this: + +**File**: `SqlScriptDom/Parser/TSql/OnOffSimpleDbOptionsHelper.cs` + +```csharp +internal bool RequiresEqualsSign(DatabaseOptionKind optionKind) +{ + switch (optionKind) + { + case DatabaseOptionKind.MemoryOptimizedElevateToSnapshot: + case DatabaseOptionKind.NestedTriggers: + case DatabaseOptionKind.TransformNoiseWords: + case DatabaseOptionKind.Ledger: + case DatabaseOptionKind.YourNewOption: // ← Add here if needed + return true; + default: + return false; + } +} +``` + +### Step 5: Create Test Files + +**Test Script**: `Test/SqlDom/TestScripts/AlterDatabaseYourOption170.sql` + +```sql +-- YourOption for VNext and Azure +ALTER DATABASE db + SET YOUR_OPTION = ON; + +ALTER DATABASE db + SET YOUR_OPTION = OFF; +``` + +**Baseline**: `Test/SqlDom/Baselines170/AlterDatabaseYourOption170.sql` + +```sql +ALTER DATABASE db + SET YOUR_OPTION = ON; + +ALTER DATABASE db + SET YOUR_OPTION = OFF; +``` + +**Test Configuration**: `Test/SqlDom/Only170SyntaxTests.cs` + +```csharp +new ParserTest170("AlterDatabaseYourOption170.sql", + nErrors80: 2, nErrors90: 2, nErrors100: 2, + nErrors110: 2, nErrors120: 2, nErrors130: 2, + nErrors140: 2, nErrors150: 2, nErrors160: 2), +``` + +### Pattern A Complete - No Ast.xml or Script Generator Changes Needed! + +--- + +## Pattern B: Custom AST Class with Sub-Options + +Use when your option has complex sub-options (e.g., `CHANGE_TRACKING (AUTO_CLEANUP = ON)`). + +### Example: ChangeTrackingDatabaseOption + +#### Step 1-3: Same as Pattern A + +Add to enum, CodeGenerationSupporter, and helpers as in Pattern A. + +#### Step 4: Define Custom AST Class + +**File**: `SqlScriptDom/Parser/TSql/Ast.xml` + +```xml + + + + + + + + + + + + + + + + + +``` + +#### Step 5: Create Custom Script Generator + +**File**: `SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.YourOptionDatabaseOption.cs` + +```csharp +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + partial class SqlScriptGeneratorVisitor + { + public override void ExplicitVisit(YourOptionDatabaseOption node) + { + GenerateIdentifier(CodeGenerationSupporter.YourOption); + GenerateSpaceAndKeyword(TSqlTokenType.EqualsSign); + GenerateSpace(); + GenerateOptionStateOnOff(node.OptionState); + + if (node.Details != null && node.Details.Count > 0) + { + GenerateSpace(); + GenerateParenthesisedCommaSeparatedList(node.Details); + } + } + + public override void ExplicitVisit(SubOption1Detail node) + { + GenerateNameEqualsValue( + CodeGenerationSupporter.SubOption1, + node.IsOn ? TSqlTokenType.On : TSqlTokenType.Off); + } + + public override void ExplicitVisit(SubOption2Detail node) + { + GenerateNameEqualsValue( + CodeGenerationSupporter.SubOption2, + node.Value); + } + } +} +``` + +#### Step 6: Update Grammar File + +**File**: `SqlScriptDom/Parser/TSql/TSql170.g` + +Add grammar rules to parse your option and sub-options. + +--- + +## Pattern C: Custom AST Class with Enum Values + +Use when your option accepts specific enum values (not just ON/OFF). + +### Example: RecoveryDatabaseOption + +#### Step 1-2: Add Enum and Keyword + +Same as Pattern A, plus define value enum: + +**File**: `SqlScriptDom/Parser/TSql/RecoveryDatabaseOptionKind.cs` + +```csharp +public enum RecoveryDatabaseOptionKind +{ + Full = 0, + BulkLogged = 1, + Simple = 2 +} +``` + +#### Step 3: Define Custom AST Class + +**File**: `SqlScriptDom/Parser/TSql/Ast.xml` + +```xml + + + + +``` + +#### Step 4: Create Helper for Values + +**File**: `SqlScriptDom/Parser/TSql/RecoveryDbOptionsHelper.cs` + +```csharp +internal class RecoveryDbOptionsHelper : OptionsHelper +{ + private RecoveryDbOptionsHelper() + { + AddOptionMapping(RecoveryDatabaseOptionKind.Full, + CodeGenerationSupporter.Full); + AddOptionMapping(RecoveryDatabaseOptionKind.BulkLogged, + CodeGenerationSupporter.BulkLogged); + AddOptionMapping(RecoveryDatabaseOptionKind.Simple, + CodeGenerationSupporter.Simple); + } + + internal static readonly RecoveryDbOptionsHelper Instance = + new RecoveryDbOptionsHelper(); +} +``` + +#### Step 5: Create Script Generator + +**File**: `SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.RecoveryDatabaseOption.cs` + +```csharp +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + partial class SqlScriptGeneratorVisitor + { + public override void ExplicitVisit(RecoveryDatabaseOption node) + { + GenerateIdentifier(CodeGenerationSupporter.Recovery); + GenerateSpace(); + RecoveryDbOptionsHelper.Instance.GenerateSourceForOption( + _writer, node.Value); + } + } +} +``` + +--- + +## Pattern D: Custom AST Class with Special Script Generation + +Use when the option has special formatting or conditional output. + +### Example: LedgerOption (Only outputs when ON) + +#### Step 1-2: Same as Pattern A + +#### Step 3: Define Custom AST Class + +**File**: `SqlScriptDom/Parser/TSql/Ast.xml` + +```xml + + + + +``` + +#### Step 4: Create Custom Script Generator + +**File**: `SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.LedgerOption.cs` + +```csharp +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + partial class SqlScriptGeneratorVisitor + { + public override void ExplicitVisit(LedgerOption node) + { + System.Diagnostics.Debug.Assert( + node.OptionKind == DatabaseOptionKind.Ledger); + + // Special behavior: Only output when ON + if (node.OptionState == OptionState.On) + { + GenerateNameEqualsValue( + CodeGenerationSupporter.Ledger, + node.OptionState.ToString()); + } + } + } +} +``` + +--- + +## Testing Checklist + +For all patterns: + +- [ ] Database option enum added to `DatabaseOptionKind.cs` +- [ ] Keyword constant added to `CodeGenerationSupporter.cs` +- [ ] Option registered in appropriate helper class +- [ ] Test script created in `TestScripts/` +- [ ] Baseline created in `Baselines/` +- [ ] Test configured in `OnlySyntaxTests.cs` +- [ ] Parser builds successfully +- [ ] Specific test passes +- [ ] **Full test suite passes** (593/593 tests) + +For Patterns B, C, D (Custom AST): + +- [ ] AST class defined in `Ast.xml` +- [ ] Custom script generator created (if needed) +- [ ] Helper class created (for enum values) +- [ ] Grammar rules updated (if new syntax) + +--- + +## Common Pitfalls + +### 1. Forgetting to Rebuild After Ast.xml Changes + +```bash +# Always rebuild after modifying Ast.xml +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug +``` + +### 2. Incorrect Error Counts in Tests + +Error count = number of statements in test script that use the new option. + +```csharp +// WRONG: Count doesn't match statements +new ParserTest170("TwoStatements.sql", nErrors160: 1) // ❌ + +// RIGHT: Match statement count +new ParserTest170("TwoStatements.sql", nErrors160: 2) // ✓ +``` + +### 3. Missing RequiresEqualsSign Registration + +Some options need `=` between name and value: +- With `=`: `LEDGER = ON`, `AUTOMATIC_INDEX_COMPACTION = ON` +- Without `=`: `AUTO_CLOSE ON`, `AUTO_SHRINK ON` + +Check existing similar options to determine which pattern to follow. + +### 4. Not Running Full Test Suite + +Grammar changes can break unrelated tests. Always run: + +```bash +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +--- + +## Examples Reference + +### Pattern A Examples (Generic ON/OFF) +- `AutoClose` +- `AutoShrink` +- `DataRetention` + +**Files to check**: +- `DatabaseOptionKind.cs` (enum value) +- `OnOffSimpleDbOptionsHelper.cs` (registration) +- No Ast.xml changes needed +- No custom script generator needed + +### Pattern B Examples (Sub-Options) +- `ChangeTrackingDatabaseOption` +- `QueryStoreDatabaseOption` +- `AutomaticTuningDatabaseOption` + +**Files to check**: +- All Pattern A files, plus: +- `Ast.xml` (custom class with Details collection) +- Custom script generator + +### Pattern C Examples (Enum Values) +- `RecoveryDatabaseOption` +- `PageVerifyDatabaseOption` +- `CursorDefaultDatabaseOption` + +**Files to check**: +- All Pattern A files, plus: +- `Ast.xml` (custom class with Value member) +- Helper class for enum mapping +- Custom script generator + +### Pattern D Examples (Special Behavior) +- `LedgerOption` (only outputs when ON) +- `OptimizedLockingDatabaseOption` (custom formatting) + +**Files to check**: +- All Pattern A files, plus: +- `Ast.xml` (custom class) +- Custom script generator with special logic + +--- + +## Quick Reference: File Locations + +| Component | Path | +|-----------|------| +| Database option enum | `SqlScriptDom/Parser/TSql/DatabaseOptionKind.cs` | +| Keyword constants | `SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs` | +| ON/OFF helper | `SqlScriptDom/Parser/TSql/OnOffSimpleDbOptionsHelper.cs` | +| Other option helpers | `SqlScriptDom/Parser/TSql/DatabaseOptionKindHelper.cs` | +| AST definitions | `SqlScriptDom/Parser/TSql/Ast.xml` | +| Script generators | `SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/` | +| Test scripts | `Test/SqlDom/TestScripts/` | +| Baselines | `Test/SqlDom/Baselines/` | +| Test classes | `Test/SqlDom/OnlySyntaxTests.cs` | + +--- + +## Version Flags Reference + +```csharp +SqlVersionFlags.TSql80 // SQL Server 2000 +SqlVersionFlags.TSql90 // SQL Server 2005 +SqlVersionFlags.TSql100 // SQL Server 2008 +SqlVersionFlags.TSql110 // SQL Server 2012 +SqlVersionFlags.TSql120 // SQL Server 2014 +SqlVersionFlags.TSql130 // SQL Server 2016 +SqlVersionFlags.TSql140 // SQL Server 2017 +SqlVersionFlags.TSql150 // SQL Server 2019 +SqlVersionFlags.TSql160 // SQL Server 2022 +SqlVersionFlags.TSql170 // SQL Server 2025 +SqlVersionFlags.TSqlFabricDW // Fabric DW + +// Combined flags +SqlVersionFlags.TSql170AndAbove // SQL 2025+ +SqlVersionFlags.TSql160AndAbove // SQL 2022+ +``` + +For Azure SQL Database, Fabric, VNext: Use the latest version (currently `TSql170AndAbove`). diff --git a/.github/instructions/debugging_workflow.guidelines.instructions.md b/.github/instructions/debugging_workflow.guidelines.instructions.md new file mode 100644 index 00000000..59932c88 --- /dev/null +++ b/.github/instructions/debugging_workflow.guidelines.instructions.md @@ -0,0 +1,252 @@ +# ScriptDOM Debugging Workflow - Quick Reference + +This is a visual guide for quickly diagnosing and fixing bugs in SqlScriptDOM. Use this as your first stop when encountering a parsing issue. + +## 🔍 Quick Diagnosis Flowchart + +``` +┌─────────────────────────────────────┐ +│ You have a parsing error/bug │ +└────────────┬────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────┐ +│ Step 1: Search for the error message │ +│ Command: grep -r "SQL46057" SqlScriptDom/ │ +│ grep -r "your error text" SqlScriptDom/ │ +└────────────┬───────────────────────────────────────────────┘ + │ + ▼ + ┌──────┴──────┐ + │ Error Type? │ + └──────┬──────┘ + │ + ┌────────┼────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────┐ ┌───────┐ ┌───────┐ +│"Option│ │"Syntax│ │Parens │ +│not │ │error │ │break │ +│valid" │ │near..." │predicte│ +└───┬───┘ └───┬───┘ └───┬───┘ + │ │ │ + ▼ ▼ ▼ + ┌─────┐ ┌─────┐ ┌─────┐ + │VALID│ │GRAM │ │PRED │ + │ATION│ │MAR │ │RECOG│ + └─────┘ └─────┘ └─────┘ +``` + +## 📋 Error Pattern Recognition + +### Pattern 1: Validation Error (Most Common) +**Symptoms:** +- ❌ `SQL46057: Option 'RESUMABLE' is not a valid index option in 'ALTER TABLE' statement` +- ❌ `Feature 'X' is not supported in SQL Server version Y` +- ✅ Same syntax works in different context (ALTER INDEX vs ALTER TABLE) + +**Quick Check:** +```bash +# Search for similar working syntax +grep -r "RESUMABLE" Test/SqlDom/TestScripts/ +# Found in AlterIndexTests but not AlterTableTests? +# → It's a validation issue! +``` + +**Solution:** [Validation_fix.guidelines.instructions.md](Validation_fix.guidelines.instructions.md) +**Files to Check:** `TSql80ParserBaseInternal.cs` (validation methods) + +--- + +### Pattern 2: Grammar Error +**Symptoms:** +- ❌ `Incorrect syntax near keyword 'NEWKEYWORD'` +- ❌ Parser doesn't recognize new T-SQL feature at all +- ❌ Syntax has never been implemented + +**Quick Check:** +```bash +# Search for the keyword in grammar files +grep -r "YourKeyword" SqlScriptDom/Parser/TSql/*.g +# Not found? → It's a grammar issue! +``` + +**Solution:** [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) +**Files to Modify:** `TSql*.g`, `Ast.xml`, Script generators + +--- + +### Pattern 3: Predicate Recognition Error +**Symptoms:** +- ✅ `WHERE REGEXP_LIKE('a', 'pattern')` works +- ❌ `WHERE (REGEXP_LIKE('a', 'pattern'))` fails with syntax error +- ❌ Error near closing parenthesis or semicolon + +**Quick Check:** +```bash +# Test both syntaxes +echo "SELECT 1 WHERE REGEXP_LIKE('a', 'b');" > test1.sql +echo "SELECT 1 WHERE (REGEXP_LIKE('a', 'b'));" > test2.sql +# Second one fails? → Predicate recognition issue! +``` + +**Solution:** [parser.guidelines.instructions.md](parser.guidelines.instructions.md) +**Files to Modify:** `TSql80ParserBaseInternal.cs` (`IsNextRuleBooleanParenthesis()`) + +--- + +## 🛠️ Standard Investigation Steps + +### Step 1: Reproduce Minimal Test Case +```bash +# Create minimal failing SQL +echo "ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);" > test.sql + +# Try parsing (use existing test harness or create simple parser test) +``` + +### Step 2: Find Error Source +```bash +# Search for error code/message +grep -r "SQL46057" SqlScriptDom/ +grep -r "is not a valid" SqlScriptDom/ + +# Common locations: +# - SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs (validation) +# - SqlScriptDom/Parser/TSql/TSql*.g (grammar rules) +# - SqlScriptDom/ScriptDom/SqlServer/*Helper.cs (option/type helpers) +``` + +### Step 3: Check Microsoft Documentation +```bash +# Search for: "Applies to: SQL Server 20XX (XX.x)" +# Verify exact version support +# Note: Different features may have different version requirements! +``` + +### Step 4: Locate Similar Working Code +```bash +# If ALTER INDEX works but ALTER TABLE doesn't: +grep -r "Resumable" Test/SqlDom/TestScripts/ + +# Find where option is registered: +grep -r "IndexOptionKind.Resumable" SqlScriptDom/ + +# Check validation paths: +grep -r "IndexAffectingStatement" SqlScriptDom/Parser/TSql/ +``` + +## 🔧 Fix Implementation Checklist + +### For Validation Fixes: +- [ ] Identify validation function (usually in `TSql80ParserBaseInternal.cs`) +- [ ] Check SQL Server version support in Microsoft docs +- [ ] Add version-gated validation (not unconditional rejection) +- [ ] Create test cases with version-specific expectations +- [ ] Build and run full test suite + +### For Grammar Fixes: +- [ ] Update grammar rules in `TSql*.g` files +- [ ] Update AST in `Ast.xml` if needed +- [ ] Update script generators +- [ ] Create test scripts and baselines +- [ ] Build parser and run tests + +### For Predicate Recognition: +- [ ] Locate `IsNextRuleBooleanParenthesis()` in `TSql80ParserBaseInternal.cs` +- [ ] Add identifier detection logic +- [ ] Add test cases with parentheses +- [ ] Verify non-parentheses syntax still works + +## 📊 Version Mapping Reference + +Quick reference for SqlVersionFlags: + +| Flag | SQL Server Version | Year | Common Features | +|------|-------------------|------|-----------------| +| TSql80AndAbove | 2000 | 2000 | Basic T-SQL | +| TSql90AndAbove | 2005 | 2005 | XML, CTEs | +| TSql100AndAbove | 2008 | 2008 | MERGE, FILESTREAM | +| TSql110AndAbove | 2012 | 2012 | Sequences, Window Functions | +| TSql120AndAbove | 2014 | 2014 | In-Memory OLTP, MAX_DURATION | +| TSql130AndAbove | 2016 | 2016 | JSON, Temporal Tables | +| TSql140AndAbove | 2017 | 2017 | Graph, STRING_AGG | +| TSql150AndAbove | 2019 | 2019 | UTF-8, Intelligent QP | +| TSql160AndAbove | 2022 | 2022 | RESUMABLE constraints, JSON improvements | +| TSql170AndAbove | 2025 | 2025 | VECTOR_SEARCH, AI features | + +## 🧪 Testing Commands Reference + +```bash +# Build parser only +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Build tests +dotnet build Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "FullyQualifiedName~YourTestName" -c Debug + +# Run specific test file pattern +dotnet test --filter "DisplayName~AlterTableResumable" -c Debug + +# Run full suite (ALWAYS do this before committing!) +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Run with detailed output +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug -v detailed +``` + +## 📂 Key Files Reference + +| File | Purpose | When to Modify | +|------|---------|---------------| +| `TSql80ParserBaseInternal.cs` | Base validation logic | Validation fixes, common logic | +| `TSql160ParserBaseInternal.cs` | Version-specific overrides | Version-specific validation | +| `TSql*.g` | Grammar rules | New syntax, grammar changes | +| `Ast.xml` | AST node definitions | New nodes, type changes | +| `IndexOptionHelper.cs` | Option registration | New options, version mappings | +| `CodeGenerationSupporter.cs` | String constants | New keywords | +| `SqlScriptGeneratorVisitor*.cs` | Script generation | Generating SQL from AST | +| `Only*SyntaxTests.cs` | Test configuration | Test expectations per version | +| `TestScripts/*.sql` | Input test cases | New test SQL | +| `Baselines*/*.sql` | Expected output | Expected formatted SQL | + +## 🚫 Common Pitfalls to Avoid + +1. **❌ Modifying shared grammar rules** → Creates unintended side effects + - ✅ Create context-specific rules instead + +2. **❌ Not running full test suite** → Breaks existing functionality + - ✅ Always run ALL 1,100+ tests before committing + +3. **❌ Assuming same version for related features** → Incorrect validation + - ✅ Check docs: MAX_DURATION (2014) ≠ RESUMABLE (2022) + +4. **❌ Forgetting script generator updates** → Round-trip fails + - ✅ Test parse → generate → parse cycle + +5. **❌ Incorrect version flag logic** → Wrong validation behavior + - ✅ Use `(flags & TSqlXXX) == 0` to check "NOT supported" + +## 🎯 Quick Decision Matrix + +| You Need To... | Use This Guide | Estimated Complexity | +|---------------|----------------|---------------------| +| Fix "option not valid" error | [Validation_fix.guidelines.instructions.md](Validation_fix.guidelines.instructions.md) | ⭐ Easy | +| Add new SQL keyword/operator | [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) | ⭐⭐⭐ Medium | +| Fix parentheses with predicates | [parser.guidelines.instructions.md](parser.guidelines.instructions.md) | ⭐⭐ Easy-Medium | +| Extend literal to expression | [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md) | ⭐⭐⭐ Medium | +| Add new statement type | [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) | ⭐⭐⭐⭐ Hard | + +## 📚 Related Documentation + +- [copilot-instructions.md](../copilot-instructions.md) - Main project documentation +- [Validation_fix.guidelines.instructions.md](Validation_fix.guidelines.instructions.md) - Version-gated validation fixes +- [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) - Grammar modifications and AST updates +- [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md) - Common extension patterns +- [parser.guidelines.instructions.md](parser.guidelines.instructions.md) - Parentheses recognition + +--- + +**Remember**: When in doubt, search for the error message first. Most bugs have been encountered before, and the error text will lead you to the right place in the code! diff --git a/.github/instructions/function.guidelines.instructions.md b/.github/instructions/function.guidelines.instructions.md new file mode 100644 index 00000000..09c0d4e2 --- /dev/null +++ b/.github/instructions/function.guidelines.instructions.md @@ -0,0 +1,363 @@ +# Guidelines for Adding New System Functions to SqlScriptDOM Parser + +This guide provides comprehensive instructions for adding new T-SQL system functions to the SqlScriptDOM parser, incorporating lessons learned from fixing JSON function parsing in RETURN statements. + +## Overview + +Adding a new system function involves three main components: +1. **AST Definition** (`Ast.xml`) - Define the abstract syntax tree node structure +2. **Grammar Rules** (`.g` files) - Define parsing logic for the function syntax +3. **Script Generator** - Handle conversion from AST back to T-SQL text +4. **Testing** - Ensure functionality works correctly across all contexts + +## Key Principle: Support Functions in RETURN Statements + +**Critical Requirement**: New system functions must be parseable in `ALTER FUNCTION` RETURN statements. This requires special handling due to ANTLR v2's limitation with semantic predicates during syntactic predicate lookahead. + +### The RETURN Statement Challenge + +The `returnStatement` grammar rule uses a syntactic predicate for lookahead: +```antlr +returnStatement: Return ((expression) => expression)? semicolonOpt +``` + +During lookahead, ANTLR cannot evaluate semantic predicates (which check runtime values like `vResult.FunctionName.Value`). This causes new functions to fail parsing in RETURN contexts even if they work elsewhere. + +## Why SELECT Works but RETURN Fails (The Core Problem) + +This section explains the fundamental issue we encountered with JSON functions and why it affects any new system function. + +### SELECT Statement Context (Always Works) +```sql +SELECT JSON_ARRAY('name'); -- ✅ Always worked +``` + +**Grammar Path**: `selectStatement` → `selectElementsList` → `selectElement` → `expression` → `expressionPrimary` → `builtInFunctionCall` + +**Why it works**: No syntactic predicates in the path - parser can evaluate semantic predicates normally during parsing. + +### RETURN Statement Context (Previously Failed) +```sql +RETURN JSON_ARRAY('name'); -- ❌ Failed before our fix +``` + +**Grammar Path**: `returnStatement` uses syntactic predicate `((expression) =>` for lookahead + +**Why it failed**: +1. Parser encounters `RETURN JSON_ARRAY(...)` +2. Syntactic predicate triggers lookahead to check if `JSON_ARRAY(...)` is a valid expression +3. During lookahead: `expression` → `expressionPrimary` → `builtInFunctionCall` +4. `builtInFunctionCall` has semantic predicate: `{(vResult.FunctionName.Value == "JSON_ARRAY")}?` +5. **ANTLR v2 limitation**: Cannot evaluate `vResult.FunctionName.Value` during lookahead (object doesn't exist yet) +6. Lookahead fails → parser assumes not an expression → syntax error + +**The Solution**: Add token-based syntactic predicates in `expressionPrimary` that work during lookahead: +```antlr +{NextTokenMatches(CodeGenerationSupporter.JsonArray) && (LA(2) == LeftParenthesis)}? +vResult=jsonArrayCall +``` + +This is why **every new system function** must include the syntactic predicate pattern to work in RETURN statements. + +## Step-by-Step Implementation Guide + +### 1. Update AST Definition (`SqlScriptDom/Parser/TSql/Ast.xml`) + +Define the function's AST node structure: + +```xml + + + + + + +``` + +**Best Practice**: Use `ScalarExpression` for parameters that should support: +- Literals (`'value'`, `123`) +- Parameters (`@param`) +- Variables (`@variable`) +- Column references (`table.column`) +- Computed expressions (`value + 1`) + +Use specific literal types only when the SQL syntax strictly requires literals. + +### 2. Add Grammar Rules (`.g` files) + +#### 2a. Define the Function Rule + +Add to the appropriate grammar files (typically `TSql160.g`, `TSql170.g`, `TSqlFabricDW.g`): + +```antlr +yourNewFunctionCall returns [YourNewFunctionCall vResult = FragmentFactory.CreateFragment()] +{ + ScalarExpression vParam1; + StringLiteral vParam2; +} + : + tFunction:Identifier LeftParenthesis + { + Match(tFunction, CodeGenerationSupporter.YourFunctionName); + UpdateTokenInfo(vResult, tFunction); + } + vParam1 = expression + { + vResult.Parameter1 = vParam1; + } + (Comma vParam2 = stringLiteral + { + vResult.Parameter2 = vParam2; + })? + RightParenthesis + ; +``` + +#### 2b. **CRITICAL**: Add Syntactic Predicate for RETURN Statement Support + +Add to `expressionPrimary` rule **before** the generic `(Identifier LeftParenthesis)` predicate: + +```antlr +expressionPrimary returns [PrimaryExpression vResult] + // ... existing rules ... + + // Add BEFORE the generic identifier predicate + | {NextTokenMatches(CodeGenerationSupporter.YourFunctionName) && (LA(2) == LeftParenthesis)}? + vResult=yourNewFunctionCall + + // ... rest of existing rules including the generic identifier case ... + | (Identifier LeftParenthesis) => vResult=builtInFunctionCall +``` + +**Why This is Required**: +- The syntactic predicate uses `NextTokenMatches()` which works during lookahead +- It must come **before** the generic `builtInFunctionCall` predicate +- This enables the function to be recognized in RETURN statements + +#### 2c. Add to Built-in Function Call (if needed) + +If your function should also be recognized through the general built-in function mechanism, add it to `builtInFunctionCall`: + +```antlr +builtInFunctionCall returns [FunctionCall vResult = FragmentFactory.CreateFragment()] + // ... existing cases ... + + | {(vResult.FunctionName.Value == "YOUR_FUNCTION_NAME")}? + vResult=yourNewFunctionCall +``` + +### 3. Update CodeGenerationSupporter Constants + +Add the function name constant to `CodeGenerationSupporter.cs`: + +```csharp +public const string YourFunctionName = "YOUR_FUNCTION_NAME"; +``` + +### 4. Create Script Generator + +Add visitor method to handle AST-to-script conversion in the appropriate script generator file: + +```csharp +public override void ExplicitVisit(YourNewFunctionCall node) +{ + GenerateIdentifier(CodeGenerationSupporter.YourFunctionName); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + + if (node.Parameter1 != null) + { + GenerateFragmentIfNotNull(node.Parameter1); + + if (node.Parameter2 != null) + { + GenerateSymbol(TSqlTokenType.Comma); + GenerateSpace(); + GenerateFragmentIfNotNull(node.Parameter2); + } + } + + GenerateSymbol(TSqlTokenType.RightParenthesis); +} +``` + +### 5. Integrate with Grammar Hierarchy + +Add your function to the appropriate place in the grammar hierarchy: + +```antlr +// Add to function call expressions +functionCall returns [FunctionCall vResult] + : vResult=yourNewFunctionCall + | // ... other function types + ; + +// Or add to primary expressions if it's a primary expression type +primaryExpression returns [PrimaryExpression vResult] + : vResult=yourNewFunctionCall + | // ... other primary expressions + ; +``` + +### 6. Build and Test + +#### 6a. Build the Project + +```bash +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug +``` + +This will regenerate parser files from the grammar. + +#### 6b. Create Test Files + +**YOU MUST ADD UNIT TESTS - DO NOT CREATE STANDALONE PROGRAMS TO TEST** + +Create test script in `Test/SqlDom/TestScripts/YourFunctionTests160.sql`: + +```sql +-- Test basic function call +SELECT YOUR_FUNCTION_NAME('param1', 'param2'); + +-- CRITICAL: Test in ALTER FUNCTION RETURN statement +ALTER FUNCTION TestYourFunction() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (YOUR_FUNCTION_NAME('value1', 'value2')); +END; +GO +``` + +#### 6c. Generate Baselines + +1. Create placeholder baseline file: `Test/SqlDom/Baselines160/YourFunctionTests160.sql` +2. Run the test (it will fail) +3. Copy the "Actual" output from the test failure +4. Update the baseline file with the correctly formatted output + +#### 6d. Configure Test + +Add test entry to `Test/SqlDom/Only160SyntaxTests.cs`: + +```csharp +new ParserTest160("YourFunctionTests160.sql", nErrors80: 1, nErrors90: 1, nErrors100: 1, nErrors110: 1, nErrors120: 1, nErrors130: 1, nErrors140: 1, nErrors150: 1), +``` + +Adjust error counts based on which SQL versions should support your function. + +#### 6e. Run Full Test Suite + +```bash +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +Ensure all tests pass, including existing ones (no regressions). + +## Real-World Example: JSON Functions Fix + +This guide incorporates lessons learned from fixing `JSON_OBJECT` and `JSON_ARRAY` parsing in RETURN statements: + +### Problem Encountered +```sql +-- This worked fine: +SELECT JSON_ARRAY('name'); -- ✅ Always worked + +-- This failed before the fix: +ALTER FUNCTION GetAuth() RETURNS NVARCHAR(MAX) AS BEGIN + RETURN (JSON_OBJECT('key': 'value')); -- ❌ Parse error here +END; +``` + +### Why SELECT Worked but RETURN Didn't + +**SELECT Statement Context** (Always Worked): +```sql +SELECT JSON_ARRAY('name'); +``` +In a SELECT statement, the parser follows this path: +1. `selectStatement` → `queryExpression` → `querySpecification` +2. `selectElementsList` → `selectElement` → `expression` +3. `expression` → `expressionPrimary` → `builtInFunctionCall` +4. ✅ No syntactic predicate blocking the path + +**RETURN Statement Context** (Previously Failed): +```sql +RETURN JSON_ARRAY('name'); +``` +In a RETURN statement, the parser follows this path: +1. `returnStatement` uses a **syntactic predicate**: `((expression) =>` +2. During lookahead, parser tries: `expression` → `expressionPrimary` → `builtInFunctionCall` +3. `builtInFunctionCall` has a **semantic predicate**: `{(vResult.FunctionName.Value == "JSON_ARRAY")}?` +4. ❌ **ANTLR v2 limitation**: Semantic predicates cannot be evaluated during syntactic predicate lookahead +5. ❌ Lookahead fails → parser doesn't recognize `JSON_ARRAY` as valid expression + +### Root Cause +The semantic predicate `{(vResult.FunctionName.Value == "JSON_OBJECT")}?` in `builtInFunctionCall` could not be evaluated during the syntactic predicate lookahead in `returnStatement`. + +### Solution Applied +Added syntactic predicates in `expressionPrimary`: + +```antlr +// Added before generic identifier predicate +| {NextTokenMatches(CodeGenerationSupporter.JsonObject) && (LA(2) == LeftParenthesis)}? + vResult=jsonObjectCall +| {NextTokenMatches(CodeGenerationSupporter.JsonArray) && (LA(2) == LeftParenthesis)}? + vResult=jsonArrayCall +``` + +This uses token-based checking (`NextTokenMatches`) which works during lookahead, unlike semantic predicates. + +## Grammar Files to Modify + +For SQL Server 2022+ functions, typically modify: +- `SqlScriptDom/Parser/TSql/TSql160.g` (SQL Server 2022) +- `SqlScriptDom/Parser/TSql/TSql170.g` (SQL Server 2025) +- `SqlScriptDom/Parser/TSql/TSqlFabricDW.g` (Azure Synapse) + +For earlier versions, add to appropriate grammar files (`TSql150.g`, `TSql140.g`, etc.). + +## Common Pitfalls + +1. **Forgetting RETURN Statement Support**: Always add syntactic predicates to `expressionPrimary` +2. **Wrong Predicate Order**: Syntactic predicates must come **before** generic predicates +3. **Semantic Predicates in Lookahead**: Don't rely on semantic predicates in contexts with syntactic predicate lookahead +4. **Missing Script Generator**: Every AST node needs a corresponding script generation visitor +5. **Incomplete Testing**: Test both standalone function calls and RETURN statement usage +6. **Version Compatibility**: Consider which SQL versions should support your function + +## Testing Checklist + +- [ ] Function parses in SELECT statements +- [ ] Function parses in WHERE clauses +- [ ] **Function parses in ALTER FUNCTION RETURN statements** +- [ ] Function parses with literal parameters +- [ ] Function parses with variable parameters +- [ ] Function parses with computed expressions as parameters +- [ ] Script generation produces correct T-SQL output +- [ ] Round-trip parsing (parse → generate → parse) works +- [ ] No regressions in existing tests +- [ ] Appropriate error handling for invalid syntax + +## Architecture Notes + +### Why Syntactic vs Semantic Predicates Matter + +- **Syntactic Predicates**: Can check token types during lookahead (`LA()`, `NextTokenMatches()`) +- **Semantic Predicates**: Check runtime values, but fail during lookahead in syntactic predicates +- **RETURN Statement Context**: Uses syntactic predicate `((expression) =>` which triggers lookahead + +### Grammar Rule Hierarchy + +``` +returnStatement + └── expression + └── expressionPrimary + ├── yourNewFunctionCall (syntactic predicate) + └── builtInFunctionCall (semantic predicate) +``` + +By adding syntactic predicates to `expressionPrimary`, we catch function calls before they reach the problematic semantic predicate in `builtInFunctionCall`. + +## Summary + +Following this guide ensures new system functions work correctly in all T-SQL contexts, especially the challenging RETURN statement scenario. The key insight is that ANTLR v2's limitations require careful predicate ordering and the use of token-based syntactic predicates for functions that need to work in lookahead contexts. \ No newline at end of file diff --git a/.github/instructions/grammar_validation.guidelines.instructions.md b/.github/instructions/grammar_validation.guidelines.instructions.md new file mode 100644 index 00000000..f02f9fea --- /dev/null +++ b/.github/instructions/grammar_validation.guidelines.instructions.md @@ -0,0 +1,359 @@ +# Validation-Based Bug Fix Guide for SqlScriptDOM + +This guide covers bugs where the **grammar already supports the syntax**, but the parser incorrectly rejects it due to validation logic. This is different from grammar-level fixes where you need to add new parsing rules. + +## When to Use This Guide + +Use this pattern when: +- ✅ The syntax **should** parse based on SQL Server documentation +- ✅ The error message is a **validation error** (e.g., "SQL46057: Option 'X' is not valid...") +- ✅ Similar syntax works in **other contexts** (e.g., ALTER INDEX works but ALTER TABLE fails) +- ✅ The feature was **added in a newer SQL Server version** but is rejected even in the correct parser + +**Do NOT use this guide when:** +- ❌ Grammar rules need to be added/modified (use [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) instead) +- ❌ AST nodes need to be created (use [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md)) +- ❌ The syntax never existed in SQL Server + +## Real-World Example: ALTER TABLE RESUMABLE Option + +### The Problem + +User reported this SQL failed to parse: +```sql +ALTER TABLE table1 +ADD CONSTRAINT PK_Constraint PRIMARY KEY CLUSTERED (a) +WITH (ONLINE = ON, MAXDOP = 2, RESUMABLE = ON, MAX_DURATION = 240); +``` + +**Error**: `SQL46057: Option 'RESUMABLE' is not a valid index option in 'ALTER TABLE' statement.` + +**But**: The same options worked fine in `ALTER INDEX` statements. + +### Investigation Steps + +#### 1. Search for the Error Message +```bash +# Search for the error code or message text +grep -r "SQL46057" SqlScriptDom/ +grep -r "is not a valid index option" SqlScriptDom/ +``` + +**Result**: Found in `TSql80ParserBaseInternal.cs` in the `VerifyAllowedIndexOption()` method. + +#### 2. Examine the Validation Logic +```csharp +// Location: SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs +protected void VerifyAllowedIndexOption(IndexAffectingStatement statement, + IndexOption option, + SqlVersionFlags versionFlags) +{ + switch (statement) + { + case IndexAffectingStatement.AlterTableAddElement: + // BEFORE: Unconditionally blocked RESUMABLE and MAX_DURATION + if (option.OptionKind == IndexOptionKind.Resumable || + option.OptionKind == IndexOptionKind.MaxDuration) + { + ThrowParseErrorException("SQL46057", /* ... */); + } + break; + // ... other cases ... + } +} +``` + +**Key Finding**: The validation was **hardcoded** to reject these options for ALTER TABLE, regardless of SQL Server version. + +#### 3. Check Microsoft Documentation +Always verify the **exact SQL Server version** support: +- **RESUMABLE**: Introduced in SQL Server 2022 (version 160) +- **MAX_DURATION**: Introduced in SQL Server 2014 (version 120) for low-priority locks, extended for resumable operations + +**Important**: Different options can have different version requirements even within the same feature set! + +### The Fix + +#### Step 1: Identify Version Flags + +The codebase uses `SqlVersionFlags` for version checking: +- `TSql80AndAbove` = SQL Server 2000+ +- `TSql90AndAbove` = SQL Server 2005+ +- `TSql100AndAbove` = SQL Server 2008+ +- `TSql110AndAbove` = SQL Server 2012+ +- `TSql120AndAbove` = SQL Server 2014+ +- `TSql130AndAbove` = SQL Server 2016+ +- `TSql140AndAbove` = SQL Server 2017+ +- `TSql150AndAbove` = SQL Server 2019+ +- `TSql160AndAbove` = SQL Server 2022+ +- `TSql170AndAbove` = SQL Server 2025+ + +#### Step 2: Apply Version-Gated Validation + +Replace unconditional rejection with version checking: + +```csharp +case IndexAffectingStatement.AlterTableAddElement: + // Invalidate RESUMABLE for versions before SQL Server 2022 (160) + // Invalidate MAX_DURATION for versions before SQL Server 2014 (120) + if (((versionFlags & SqlVersionFlags.TSql160AndAbove) == 0 && + option.OptionKind == IndexOptionKind.Resumable) || + ((versionFlags & SqlVersionFlags.TSql120AndAbove) == 0 && + option.OptionKind == IndexOptionKind.MaxDuration)) + { + // Throw an error indicating the option is not supported in the current SQL Server version + ThrowParseErrorException("SQL46057", "Option not supported in this SQL Server version."); + } + break; +``` + +**Pattern Explanation**: +- `(versionFlags & SqlVersionFlags.TSql160AndAbove) == 0` → Returns true if parser version < 160 +- If true AND option is RESUMABLE → Throw error (option not supported yet) +- Same pattern for MAX_DURATION with TSql120AndAbove + +#### Step 3: Create Comprehensive Tests + +**Test Script**: `Test/SqlDom/TestScripts/AlterTableResumableTests160.sql` +```sql +-- Test 1: RESUMABLE with MAX_DURATION (minutes) +ALTER TABLE dbo.MyTable +ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) +WITH (RESUMABLE = ON, MAX_DURATION = 240 MINUTES); + +-- Test 2: RESUMABLE = ON +ALTER TABLE dbo.MyTable +ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) +WITH (RESUMABLE = ON); + +-- Test 3: RESUMABLE = OFF +ALTER TABLE dbo.MyTable +ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) +WITH (RESUMABLE = OFF); + +-- Test 4: UNIQUE constraint with RESUMABLE +ALTER TABLE dbo.MyTable +ADD CONSTRAINT uq_test UNIQUE NONCLUSTERED (name) +WITH (RESUMABLE = ON); +``` + +#### Step 4: Configure Test Expectations + +**Test Configuration**: `Test/SqlDom/Only160SyntaxTests.cs` +```csharp +new ParserTest160("AlterTableResumableTests160.sql"), +``` + +#### Step 5: Create Baseline Files + +**Baseline**: `Test/SqlDom/Baselines160/AlterTableResumableTests160.sql` +```sql +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = ON, MAX_DURATION = 240 MINUTES); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = ON); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT pk_test PRIMARY KEY CLUSTERED (id) WITH (RESUMABLE = OFF); + +ALTER TABLE dbo.MyTable + ADD CONSTRAINT uq_test UNIQUE NONCLUSTERED (name) WITH (RESUMABLE = ON); +``` + +#### Step 6: Validate the Fix + +```bash +# Build to ensure code compiles +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Run specific test +dotnet test --filter "FullyQualifiedName~AlterTableResumableTests" -c Debug + +# Run FULL test suite to catch regressions +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +**Expected Results**: +- ✅ TSql160Parser: 0 errors (all tests pass) +- ✅ TSql80-150Parsers: 4 errors each (RESUMABLE correctly rejected) +- ✅ Full suite: 1,116 tests passed, 0 failed + +## Common Validation Patterns + +### Pattern 1: Version-Gated Validation +```csharp +// Allow feature only in specific SQL Server versions +if ((versionFlags & SqlVersionFlags.TSqlXXXAndAbove) == 0 && + condition) +{ + ThrowParseErrorException(...); +} +``` + +### Pattern 2: Multiple Version Requirements +```csharp +// Different features with different version requirements +if (((versionFlags & SqlVersionFlags.TSql160AndAbove) == 0 && feature1) || + ((versionFlags & SqlVersionFlags.TSql120AndAbove) == 0 && feature2)) +{ + ThrowParseErrorException(...); +} +``` + +### Pattern 3: Context-Specific Validation +```csharp +// Same option, different rules for different statements +switch (statement) +{ + case IndexAffectingStatement.AlterTableAddElement: + // Stricter rules for ALTER TABLE + break; + case IndexAffectingStatement.CreateIndex: + // More permissive for CREATE INDEX + break; +} +``` + +## Key Files for Validation Fixes + +### 1. Validation Logic +- **`SqlScriptDom/Parser/TSql/TSql80ParserBaseInternal.cs`** + - Base validation shared by all parser versions + - Contains `VerifyAllowedIndexOption()`, `VerifyAllowedIndexType()`, etc. + - Most validation fixes happen here + +### 2. Version-Specific Overrides +- **`SqlScriptDom/Parser/TSql/TSql160ParserBaseInternal.cs`** + - Can override base validation for specific versions + - Example: `VerifyAllowedIndexOption160()` calls base then adds version-specific logic + +### 3. Option Registration +- **`SqlScriptDom/ScriptDom/SqlServer/IndexOptionHelper.cs`** + - Maps option keywords to `IndexOptionKind` enum values + - Defines version support: `AddOptionMapping(kind, keyword, versionFlags)` + - **Note**: Registration here controls grammar acceptance, validation happens separately + +### 4. Enums and Constants +- **`SqlScriptDom/ScriptDom/SqlServer/IndexAffectingStatement.cs`** + - Defines statement types: `CreateIndex`, `AlterIndex`, `AlterTableAddElement`, etc. + +- **`SqlScriptDom/ScriptDom/SqlServer/IndexOptionKind.cs`** + - Defines option types: `Resumable`, `MaxDuration`, `Online`, etc. + +## Debugging Workflow + +### Step 1: Reproduce the Error +```bash +# Create a minimal test file +echo "ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id) WITH (RESUMABLE = ON);" > test.sql + +# Try parsing it (will fail) +# Use your test harness or create a simple parser test +``` + +### Step 2: Find the Error Source +```bash +# Search for error code +grep -r "SQL46057" SqlScriptDom/ + +# Search for error message text +grep -r "is not a valid" SqlScriptDom/ +``` + +### Step 3: Locate Validation Function +Common validation functions to check: +- `VerifyAllowedIndexOption()` - Most common +- `VerifyAllowedIndexType()` +- `VerifyFeatureSupport()` +- `CheckFeatureAvailability()` + +### Step 4: Examine the Logic +Look for: +- Hardcoded rejections (unconditional throws) +- Version checks that are too strict +- Missing version flag checks +- Incorrect version constants + +### Step 5: Check Similar Working Cases +If ALTER INDEX works but ALTER TABLE doesn't: +- Compare their validation paths +- Check for different `switch` cases +- Look for statement-type specific logic + +## Testing Strategy + +### Test Coverage Checklist +- [ ] Test with option enabled (`OPTION = ON`) +- [ ] Test with option disabled (`OPTION = OFF`) +- [ ] Test with option + other options (`OPTION = ON, OTHER_OPTION = value`) +- [ ] Test different statement types (PRIMARY KEY, UNIQUE, etc.) +- [ ] Test across all SQL Server versions (verify error counts) + +### Version-Specific Error Expectations +```csharp +// Pattern for test configuration +new ParserTestXXX("TestFile.sql", + nErrors80: X, // Count errors for SQL 2000 + nErrors90: X, // Count errors for SQL 2005 + nErrors100: X, // Count errors for SQL 2008 + nErrors110: X, // Count errors for SQL 2012 + nErrors120: Y, // May differ if feature added in 2014 + nErrors130: Y, // Same as above + nErrors140: Y, // Same as above + nErrors150: Y, // Same as above + // nErrors160: 0 (implicit) - Feature supported in 2022+ +) +``` + +## Common Pitfalls + +### 1. Assuming Same Version for Related Features +❌ **Wrong**: "RESUMABLE and MAX_DURATION are both resumable features, so both need TSql160+" +✅ **Correct**: Check documentation - MAX_DURATION existed before RESUMABLE (TSql120 vs TSql160) + +### 2. Not Running Full Test Suite +❌ **Wrong**: Only run the new test, assume it's fine +✅ **Correct**: Run ALL tests - validation changes can affect unexpected areas + +### 3. Incorrect Version Flag Logic +❌ **Wrong**: `if (versionFlags & SqlVersionFlags.TSql160AndAbove)` (missing == 0) +✅ **Correct**: `if ((versionFlags & SqlVersionFlags.TSql160AndAbove) == 0)` (check if NOT set) + +### 4. Forgetting Statement Context +❌ **Wrong**: Apply same validation to all statement types +✅ **Correct**: Different statements may have different option support + +## Summary Checklist + +- [ ] **Identify** the validation function throwing the error +- [ ] **Verify** Microsoft documentation for exact version support +- [ ] **Apply** version-gated validation (not unconditional rejection) +- [ ] **Create** comprehensive test cases covering all scenarios +- [ ] **Configure** test expectations for all SQL Server versions +- [ ] **Generate** baseline files from actual parser output +- [ ] **Build** the ScriptDOM project successfully +- [ ] **Run** full test suite (ALL 1,100+ tests must pass) +- [ ] **Document** the fix with clear before/after examples + +## Related Guides + +- [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) - For grammar-level fixes +- [grammer.guidelines.instructions.md](grammer.guidelines.instructions.md) - For extending existing grammar +- [parser.guidelines.instructions.md](parser.guidelines.instructions.md) - For parentheses recognition issues + +## Real-World Examples + +### Example 1: ALTER TABLE RESUMABLE (SQL Server 2022) +- **File**: `TSql80ParserBaseInternal.cs` +- **Function**: `VerifyAllowedIndexOption()` +- **Fix**: Added `TSql160AndAbove` check for RESUMABLE +- **Tests**: `AlterTableResumableTests160.sql` + +### Example 2: MAX_DURATION (SQL Server 2014) +- **File**: Same as above +- **Function**: Same as above +- **Fix**: Added `TSql120AndAbove` check for MAX_DURATION +- **Tests**: Same file, different version expectations + +These examples demonstrate how validation fixes are often simpler than grammar changes - the parser already knows how to parse the syntax, it just needs permission to accept it in specific contexts and versions. diff --git a/.github/GRAMMAR_EXTENSION_PATTERNS.md b/.github/instructions/grammer.guidelines.instructions.md similarity index 88% rename from .github/GRAMMAR_EXTENSION_PATTERNS.md rename to .github/instructions/grammer.guidelines.instructions.md index 900026f8..53311f7d 100644 --- a/.github/GRAMMAR_EXTENSION_PATTERNS.md +++ b/.github/instructions/grammer.guidelines.instructions.md @@ -64,17 +64,32 @@ yourContextParameterRule returns [ScalarExpression vResult] Most script generators using `GenerateNameEqualsValue()` or similar methods work automatically with `ScalarExpression`. No changes typically needed. #### Step 4: Add Test Coverage -```sql --- Test parameter -FUNCTION_NAME(PARAM = @parameter) - --- Test outer reference -FUNCTION_NAME(PARAM = outerref.column) - --- Test computed expression -FUNCTION_NAME(PARAM = value + 1) +Add tests within the existing test framework: +```csharp +[TestMethod] +public void VerifyGrammarExtension() +{ + var parser = new TSql170Parser(true); + + // Test parameter + var sql1 = "SELECT FUNCTION_NAME(PARAM = @parameter)"; + var result1 = parser.Parse(new StringReader(sql1), out var errors1); + Assert.AreEqual(0, errors1.Count, "Parameter syntax should work"); + + // Test outer reference + var sql2 = "SELECT FUNCTION_NAME(PARAM = outerref.column)"; + var result2 = parser.Parse(new StringReader(sql2), out var errors2); + Assert.AreEqual(0, errors2.Count, "Outer reference syntax should work"); + + // Test computed expression + var sql3 = "SELECT FUNCTION_NAME(PARAM = value + 1)"; + var result3 = parser.Parse(new StringReader(sql3), out var errors3); + Assert.AreEqual(0, errors3.Count, "Computed expression syntax should work"); +} ``` +**⚠️ CRITICAL**: Add this test method to an existing test class (e.g., `Only170SyntaxTests.cs`). **Never create standalone test projects.** + ### Real-World Example: VECTOR_SEARCH TOP_N **Problem**: `VECTOR_SEARCH` TOP_N parameter only accepted integer literals. diff --git a/.github/instructions/new_data_types.guidelines.instructions.md b/.github/instructions/new_data_types.guidelines.instructions.md new file mode 100644 index 00000000..86e59a67 --- /dev/null +++ b/.github/instructions/new_data_types.guidelines.instructions.md @@ -0,0 +1,439 @@ +# Guidelines for Adding New Data Types to SqlScriptDOM + +This guide provides step-by-step instructions for adding support for completely new SQL Server data types to the SqlScriptDOM parser. This pattern was established from the Vector data type implementation (commits 38a0971 and cd69b78). + +## When to Use This Guide + +Use this pattern when: +- ✅ Adding a **completely new SQL Server data type** (e.g., VECTOR, GEOMETRY, GEOGRAPHY) +- ✅ The data type has **custom parameters** not handled by standard SQL data types +- ✅ The data type requires **specialized parsing logic** beyond simple name/size parameters +- ✅ The data type is **introduced in a specific SQL Server version** + +**Do NOT use this guide for:** +- ❌ Modifying existing data types (use [validation_fix.guidelines.instructions.md](validation_fix.guidelines.instructions.md)) +- ❌ Adding function syntax (use [function.guidelines.instructions.md](function.guidelines.instructions.md)) +- ❌ Simple keyword additions (use [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md)) + +## Real-World Example: Vector Data Type + +The Vector data type implementation demonstrates this pattern: + +### SQL Server Syntax Supported +```sql +-- Basic vector with dimension only +DECLARE @embedding AS VECTOR(1536); +CREATE TABLE tbl (embedding VECTOR(1536)); + +-- Vector with dimension and base type +DECLARE @embedding AS VECTOR(1536, FLOAT32); +CREATE TABLE tbl (embedding VECTOR(1536, FLOAT16)); +``` + +### Key Challenge Solved +The Vector type requires custom parsing because: +- **Standard data types** use size parameters: `VARCHAR(50)`, `DECIMAL(10,2)` +- **Vector type** uses dimension + optional base type: `VECTOR(1536, FLOAT32)` +- **Base type parameter** is an identifier (FLOAT16/FLOAT32), not a size literal + +## Step-by-Step Implementation Guide + +### 1. Define AST Node Structure (`Ast.xml`) + +Add a new class inheriting from `DataTypeReference`: + +```xml + + + + + + + + + +``` + +**Design Principles**: +- **Inherit from `DataTypeReference`**: All SQL data types inherit from this base class +- **Choose appropriate member types**: + - `IntegerLiteral`: For numeric parameters (dimensions, sizes) + - `Identifier`: For type names or keywords + - `StringLiteral`: For string parameters + - `ScalarExpression`: For complex expressions (use sparingly) +- **Optional parameters**: Members can be null for optional syntax + +### 2. Add Grammar Rule (`TSql*.g`) + +Create a specialized parsing rule for your data type: + +```antlr +// Location: SqlScriptDom/Parser/TSql/TSql170.g (or appropriate version) +// Add after xmlDataType rule (~line 30672) + +yourDataType [SchemaObjectName vName] returns [YourDataTypeReference vResult = FragmentFactory.CreateFragment()] +{ + vResult.Name = vName; + vResult.UpdateTokenInfo(vName); + + IntegerLiteral vParameter1 = null; + Identifier vParameter2 = null; +} + : + ( LeftParenthesis vParameter1=integer + { + vResult.Parameter1 = vParameter1; + } + ( + Comma vParameter2=identifier + { + vResult.Parameter2 = vParameter2; + } + )? + tRParen:RightParenthesis + { + UpdateTokenInfo(vResult,tRParen); + } + ) + ; +``` + +**Grammar Pattern Explanation**: +- **Function signature**: Takes `SchemaObjectName vName` parameter and returns your AST type +- **Variable declarations**: Declare variables for each parameter using appropriate types +- **Parameter parsing**: Use `integer`, `identifier`, `stringLiteral` based on syntax needs +- **Optional parameters**: Wrap in `( ... )?` syntax for optional elements +- **Token info updates**: Always call `UpdateTokenInfo()` for proper source location tracking + +### 3. Integrate with Scalar Data Type Rule + +Connect your new grammar rule to the main data type parsing logic: + +```antlr +// Location: SqlScriptDom/Parser/TSql/TSql170.g +// Find scalarDataType rule (~line 30694) and add your type check + +scalarDataType returns [DataTypeReference vResult = null] +{ + SchemaObjectName vName; + SqlDataTypeOption typeOption = SqlDataTypeOption.None; + // ... existing variables ... +} + : vName = schemaObjectFourPartName + { + typeOption = GetSqlDataTypeOption(vName); + // ... existing logic ... + } + ( + ( + {isXmlDataType}? + vResult = xmlDataType[vName] + | + {typeOption == SqlDataTypeOption.YourType}? // Add this condition + vResult = yourDataType[vName] + | + {typeOption != SqlDataTypeOption.None}? + vResult = sqlDataTypeWithoutNational[vName, typeOption] + // ... rest of existing alternatives +``` + +**Integration Requirements**: +- **Add type option check**: Use `{typeOption == SqlDataTypeOption.YourType}?` semantic predicate +- **Maintain order**: Place before the generic `sqlDataTypeWithoutNational` fallback +- **Update SqlDataTypeOption enum**: Add your type to the enum (implementation dependent) + +### 4. Add String Constants + +Add necessary string constants for keywords: + +```csharp +// Location: SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs +// Add alphabetically in the constants section (~line 427 for Float constants) + +internal const string YourType = "YOURTYPE"; +internal const string YourTypeParam1 = "PARAM1_KEYWORD"; +internal const string YourTypeParam2 = "PARAM2_KEYWORD"; +``` + +**Naming Convention**: +- **Use exact SQL keyword casing**: `VECTOR`, `FLOAT16`, `FLOAT32` +- **Group related constants**: Keep data type constants together +- **Alphabetical ordering**: Maintain alphabetical order within sections + +### 5. Create Script Generator + +Implement the visitor method to convert AST back to T-SQL: + +```csharp +// Location: SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.YourDataType.cs +// Create new file following naming convention + +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ + +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + partial class SqlScriptGeneratorVisitor + { + public override void ExplicitVisit(YourDataTypeReference node) + { + GenerateIdentifier(CodeGenerationSupporter.YourType); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + GenerateFragmentIfNotNull(node.Parameter1); + if (node.Parameter2 != null) + { + GenerateSymbol(TSqlTokenType.Comma); + GenerateSpaceAndFragmentIfNotNull(node.Parameter2); + } + GenerateSymbol(TSqlTokenType.RightParenthesis); + } + } +} +``` + +**Script Generation Patterns**: +- **Use `GenerateIdentifier()`**: For type names and keywords +- **Use `GenerateSymbol()`**: For punctuation (`LeftParenthesis`, `Comma`, etc.) +- **Use `GenerateFragmentIfNotNull()`**: For required parameters +- **Use `GenerateSpaceAndFragmentIfNotNull()`**: For optional parameters with preceding space +- **Handle optional parameters**: Always check for null before generating + +### 6. Build and Test Grammar Changes + +Build the project to regenerate parser files: + +```bash +# Build the ScriptDOM project to regenerate parser from grammar +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug +``` + +**Common Build Issues**: +- **Grammar syntax errors**: Check ANTLR syntax in `.g` files +- **Missing constants**: Ensure all referenced constants exist in `CodeGenerationSupporter.cs` +- **AST node mismatches**: Verify AST class names match grammar return types + +### 7. Create Comprehensive Test Scripts + +Create test script covering all syntax variations: + +```sql +-- File: Test/SqlDom/TestScripts/YourDataTypeTests170.sql + +-- Basic syntax with single parameter +CREATE TABLE tbl (col1 YOURTYPE(100)); +DECLARE @var1 AS YOURTYPE(100); + +-- Extended syntax with optional parameters +CREATE TABLE tbl (col1 YOURTYPE(100, PARAM1)); +DECLARE @var2 AS YOURTYPE(100, PARAM2); + +-- Case insensitivity testing +CREATE TABLE tbl (col1 yourtype(100, param1)); +DECLARE @var3 AS YOURTYPE(100, param2); + +-- Integration with other SQL constructs +CREATE TABLE tbl ( + id INT PRIMARY KEY, + data YOURTYPE(100, PARAM1) NOT NULL +); + +-- Variables and parameters +CREATE FUNCTION TestFunction(@input YOURTYPE(100)) +RETURNS YOURTYPE(200, PARAM2) +AS +BEGIN + DECLARE @result YOURTYPE(200, PARAM2); + RETURN @result; +END; +``` + +**Test Coverage Requirements**: +- **All syntax variations**: Test with and without optional parameters +- **Case sensitivity**: Test different case combinations +- **Integration contexts**: Variables, table columns, function parameters/returns +- **Edge cases**: Minimum/maximum parameter values if applicable + +### 8. Generate Baseline Files + +Create the expected output baseline: + +1. **Create placeholder baseline**: `Test/SqlDom/Baselines170/YourDataTypeTests170.sql` +2. **Run the test** (will fail initially): + ```bash + dotnet test --filter "YourDataTypeTests170" Test/SqlDom/UTSqlScriptDom.csproj -c Debug + ``` +3. **Copy "Actual" output** from test failure into baseline file +4. **Verify formatting** matches parser's standard formatting + +**Baseline Example** (Vector data type): +```sql +CREATE TABLE tbl ( + embedding VECTOR(1) +); + +CREATE TABLE tbl ( + embedding VECTOR(1, float32) +); + +DECLARE @embedding AS VECTOR(2); + +DECLARE @embedding AS VECTOR(2, FLOAT32); +``` + +### 9. Configure Test Expectations + +Add test configuration to version-specific test class: + +```csharp +// Location: Test/SqlDom/Only170SyntaxTests.cs (or appropriate version) +// Add to the ParserTest170 array + +new ParserTest170("YourDataTypeTests170.sql"), +``` + +**Error Count Guidelines**: +- **Count all syntax instances**: Each usage of the new type in test script +- **Consider SQL version support**: When was the feature actually introduced? +- **Consistent across versions**: Usually same error count until supported version +- **Test validation**: Run tests to verify error counts are accurate + +### 10. Full Test Suite Validation + +Run complete test suite to ensure no regressions: + +```bash +# Run all ScriptDOM tests +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Expected result: All tests pass, including new data type tests +# Total tests: 1,100+ (number increases with new features) +``` + +**Regression Prevention**: +- **Grammar changes can break existing functionality**: Shared rules affect multiple contexts +- **AST changes can break script generation**: Ensure all visitors are updated +- **Version compatibility**: New syntax shouldn't break older version parsers + +## Advanced Considerations + +### Version-Specific Implementation + +For data types introduced in specific SQL Server versions: + +```antlr +// Different grammar files for different SQL versions +// TSql160.g - SQL Server 2022 features +// TSql170.g - SQL Server 2025 features +// TSqlFabricDW.g - Azure Synapse features +``` + +**Guidelines**: +- **Target appropriate version**: Add to the SQL version where feature was introduced +- **Cascade to later versions**: Copy rules to all subsequent version grammar files +- **Version-specific testing**: Test error behavior in older parsers + +### Complex Parameter Types + +For data types requiring complex parameter parsing: + +```xml + + + +``` + +**When to use complex types**: +- **ScalarExpression**: When parameters can be variables, function calls, or computed values +- **Collections**: When syntax supports multiple values or options +- **Custom classes**: When parameters have their own sub-syntax + +### Script Generator Considerations + +For complex formatting requirements: + +```csharp +public override void ExplicitVisit(ComplexDataTypeReference node) +{ + GenerateIdentifier(CodeGenerationSupporter.ComplexType); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + + // Complex formatting with line breaks + if (node.HasMultipleParameters) + { + Indent(); + GenerateNewLine(); + } + + GenerateCommaSeparatedList(node.Parameters); + + if (node.HasMultipleParameters) + { + Outdent(); + GenerateNewLine(); + } + + GenerateSymbol(TSqlTokenType.RightParenthesis); +} +``` + +## Common Pitfalls and Solutions + +### 1. Forgetting Script Generator Implementation +**Problem**: AST node created but no script generation visitor +**Solution**: Always implement `ExplicitVisit()` method for new AST nodes + +### 2. Incorrect Grammar Integration +**Problem**: Data type not recognized in all contexts +**Solution**: Ensure integration with `scalarDataType` rule and proper semantic predicates + +### 3. Missing Version Compatibility +**Problem**: New type breaks older version parsers unexpectedly +**Solution**: Add proper version checks and test all SQL Server versions + +### 4. Incomplete Test Coverage +**Problem**: Edge cases not covered in testing +**Solution**: Test all syntax variations, case sensitivity, and integration contexts + +### 5. AST Design Issues +**Problem**: AST doesn't properly represent the SQL syntax +**Solution**: Design AST members to match SQL parameter structure and optionality + +## Validation Checklist + +- [ ] **AST Definition**: New class inherits from correct base class with appropriate members +- [ ] **Grammar Rules**: Specialized parsing rule handles all syntax variations +- [ ] **Grammar Integration**: Connected to `scalarDataType` with proper semantic predicate +- [ ] **String Constants**: All keywords added to `CodeGenerationSupporter.cs` +- [ ] **Script Generator**: `ExplicitVisit()` method generates correct T-SQL output +- [ ] **Test Scripts**: Comprehensive test coverage including edge cases +- [ ] **Baseline Files**: Generated output matches expected formatted T-SQL +- [ ] **Test Configuration**: Error counts configured for all SQL Server versions +- [ ] **Build Success**: Project builds without errors and regenerates parser +- [ ] **Full Test Suite**: All existing tests continue to pass (no regressions) + +## Related Guides + +- [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) - For general grammar modifications +- [function.guidelines.instructions.md](function.guidelines.instructions.md) - For adding system functions +- [validation_fix.guidelines.instructions.md](validation_fix.guidelines.instructions.md) - For validation-only issues +- [testing.guidelines.instructions.md](testing.guidelines.instructions.md) - For comprehensive testing strategies + +## Real-World Examples + +### Vector Data Type (SQL Server 2025) +- **AST Class**: `VectorDataTypeReference` with `Dimension` and `BaseType` members +- **Syntax**: `VECTOR(1536)`, `VECTOR(1536, FLOAT32)` +- **Commits**: cd69b78, 38a0971 +- **Challenge**: Optional second parameter with identifier type + +### Future Examples +This pattern can be applied to other SQL Server data types like: +- **GEOMETRY**: Spatial data with complex parameters +- **GEOGRAPHY**: Geographic data with coordinate systems +- **HIERARCHYID**: Hierarchical data with custom syntax +- **Custom CLR Types**: User-defined types with specialized parameters + +The Vector implementation serves as the canonical example for this pattern and should be referenced for similar future data type additions. \ No newline at end of file diff --git a/.github/instructions/new_index_types.guidelines.instructions.md b/.github/instructions/new_index_types.guidelines.instructions.md new file mode 100644 index 00000000..dacba71c --- /dev/null +++ b/.github/instructions/new_index_types.guidelines.instructions.md @@ -0,0 +1,546 @@ +# Guidelines for Adding New Index Types to SqlScriptDOM + +This guide provides step-by-step instructions for adding support for new SQL Server index types to the SqlScriptDOM parser. This pattern was established from the JSON and Vector index implementations found in SQL Server 2025 (TSql170). + +## When to Use This Guide + +Use this pattern when: +- ✅ Adding a **completely new SQL Server index type** (e.g., JSON INDEX, VECTOR INDEX, SPATIAL INDEX) +- ✅ The index type has **specialized syntax** not handled by standard CREATE INDEX +- ✅ The index type requires **custom parsing logic** for type-specific clauses or options +- ✅ The index type is **introduced in a specific SQL Server version** + +**Do NOT use this guide for:** +- ❌ Adding new index options to existing index types (use [validation_fix.guidelines.instructions.md](validation_fix.guidelines.instructions.md)) +- ❌ Adding standard indexes with new keywords (use [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md)) +- ❌ Adding function or data type syntax (use respective guides) + +## Real-World Examples: JSON and Vector Indexes + +### JSON Index Implementation +```sql +-- Basic JSON index +CREATE JSON INDEX IX_JSON_Basic ON dbo.Users (JsonData); + +-- JSON index with FOR clause (multiple paths) +CREATE JSON INDEX IX_JSON_Paths ON dbo.Users (JsonData) +FOR ('$.name', '$.email', '$.age'); + +-- JSON index with WITH options +CREATE JSON INDEX IX_JSON_Options ON dbo.Users (JsonData) +WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON, MAXDOP = 4); +``` + +### Vector Index Implementation +```sql +-- Basic vector index +CREATE VECTOR INDEX IX_Vector_Basic ON dbo.Documents (VectorData); + +-- Vector index with metric and type +CREATE VECTOR INDEX IX_Vector_Complete ON dbo.Documents (VectorData) +WITH (METRIC = 'cosine', TYPE = 'DiskANN'); + +-- Vector index with filegroup +CREATE VECTOR INDEX IX_Vector_FG ON dbo.Documents (VectorData) +WITH (METRIC = 'dot') +ON [PRIMARY]; +``` + +### Key Challenges Solved +- **Type-specific syntax**: JSON INDEX has `FOR (paths)` clause, VECTOR INDEX has `METRIC`/`TYPE` options +- **Custom columns**: Single column specification instead of column lists +- **Specialized options**: New index options specific to each index type +- **Grammar integration**: Seamless integration with existing CREATE INDEX patterns + +## Step-by-Step Implementation Guide + +### 1. Define AST Node Structure (`Ast.xml`) + +Add a new class inheriting from `IndexStatement`: + +```xml + + + + + + + + + + + +``` + +**Design Principles**: +- **Inherit from `IndexStatement`**: All index types inherit from this base class +- **Reuse standard properties**: `Name`, `OnName`, `IndexOptions` come from base class +- **Add type-specific members**: Properties unique to your index type +- **Collections for lists**: Use `Collection="true"` for array-like properties +- **Optional members**: Members can be null for optional syntax elements + +### 2. Add Grammar Rule (`TSql*.g`) + +Create a specialized parsing rule for your index type: + +```antlr +// Location: SqlScriptDom/Parser/TSql/TSql170.g (or appropriate version) +// Add after existing index statement rules (~line 17021) + +createYourTypeIndexStatement [IToken tUnique, bool? isClustered] returns [CreateYourTypeIndexStatement vResult = FragmentFactory.CreateFragment()] +{ + Identifier vIdentifier; + SchemaObjectName vSchemaObjectName; + Identifier vSpecializedColumn; + StringLiteral vProperty; + FileGroupOrPartitionScheme vFileGroupOrPartitionScheme; + + if (tUnique != null) + { + ThrowIncorrectSyntaxErrorException(tUnique); + } + if (isClustered.HasValue) + { + ThrowIncorrectSyntaxErrorException(LT(1)); + } +} + : tYourType:Identifier tIndex:Index vIdentifier=identifier + { + Match(tYourType, CodeGenerationSupporter.YourType); + vResult.Name = vIdentifier; + } + tOn:On vSchemaObjectName=schemaObjectThreePartName + { + vResult.OnName = vSchemaObjectName; + } + LeftParenthesis vSpecializedColumn=identifier tRParen:RightParenthesis + { + vResult.SpecializedColumn = vSpecializedColumn; + UpdateTokenInfo(vResult, tRParen); + } + ( + tFor:For LeftParenthesis + vProperty=stringLiteral + { + AddAndUpdateTokenInfo(vResult, vResult.TypeSpecificProperty, vProperty); + } + ( + Comma vProperty=stringLiteral + { + AddAndUpdateTokenInfo(vResult, vResult.TypeSpecificProperty, vProperty); + } + )* + RightParenthesis + )? + ( + // Greedy due to conflict with withCommonTableExpressionsAndXmlNamespaces + options {greedy = true; } : + With + indexOptionList[IndexAffectingStatement.CreateIndex, vResult.IndexOptions, vResult] + )? + ( + On vFileGroupOrPartitionScheme=filegroupOrPartitionScheme + { + vResult.OnFileGroupOrPartitionScheme = vFileGroupOrPartitionScheme; + } + )? + ; +``` + +**Grammar Pattern Explanation**: +- **Parameter validation**: Reject UNIQUE and CLUSTERED if not supported +- **Standard index structure**: Name, ON table, column specification +- **Type-specific clauses**: Optional FOR, WITH, ON clauses as appropriate +- **Token matching**: Use `Match()` to verify keywords +- **Collection building**: Use `AddAndUpdateTokenInfo()` for lists + +### 3. Integrate with Main Index Grammar + +Add your index type to the main CREATE INDEX rule: + +```antlr +// Location: SqlScriptDom/Parser/TSql/TSql170.g +// Find createIndexStatement rule (~line 16880) and add integration + +createIndexStatement [IToken tUnique, bool? isClustered] returns [TSqlStatement vResult] + : // ... existing alternatives ... + | vResult=createYourTypeIndexStatement[tUnique, isClustered] + ; + +// Also add to ddlStatement if needed (~line 885) +ddlStatement returns [TSqlStatement vResult] + : // ... existing alternatives ... + | vResult=createYourTypeIndexStatement[null, null] + // ... rest of alternatives ... + ; +``` + +**Integration Requirements**: +- **Add to `createIndexStatement`**: Main CREATE INDEX dispatch rule +- **Add to `ddlStatement`**: Top-level DDL statement recognition +- **Parameter passing**: Pass `tUnique` and `isClustered` tokens +- **Consistent ordering**: Place appropriately among other index types + +### 4. Add String Constants + +Add necessary keywords to `CodeGenerationSupporter.cs`: + +```csharp +// Location: SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs +// Add alphabetically in the constants section + +internal const string YourType = "YOUR_TYPE"; +internal const string YourTypeSpecificKeyword1 = "KEYWORD1"; +internal const string YourTypeSpecificKeyword2 = "KEYWORD2"; +``` + +**Naming Convention**: +- **Use exact SQL keyword casing**: `Json`, `Vector`, `Metric` +- **Group related constants**: Keep index-specific constants together +- **Follow existing patterns**: Match existing naming conventions + +### 5. Add Index Options (if needed) + +If your index type requires new index options, add them: + +```csharp +// Location: SqlScriptDom/ScriptDom/SqlServer/IndexOptionKind.cs +// Add to the enum +public enum IndexOptionKind +{ + // ... existing options ... + YourTypeSpecificOption1, + YourTypeSpecificOption2, + // ... rest of enum ... +} + +// Location: SqlScriptDom/Parser/TSql/IndexOptionHelper.cs +// Add option mappings in the constructor +AddOptionMapping(IndexOptionKind.YourTypeSpecificOption1, CodeGenerationSupporter.YourTypeSpecificKeyword1, SqlVersionFlags.TSql170AndAbove); +AddOptionMapping(IndexOptionKind.YourTypeSpecificOption2, CodeGenerationSupporter.YourTypeSpecificKeyword2, SqlVersionFlags.TSql170AndAbove); +``` + +**Option Guidelines**: +- **Add to enum**: Define new `IndexOptionKind` values +- **Register mappings**: Map keywords to enum values with version flags +- **Version compatibility**: Use appropriate `SqlVersionFlags` + +### 6. Create Script Generator + +Implement the visitor method to convert AST back to T-SQL: + +```csharp +// Location: SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.CreateYourTypeIndexStatement.cs +// Create new file following naming convention + +//------------------------------------------------------------------------------ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// +//------------------------------------------------------------------------------ +using System.Collections.Generic; +using Microsoft.SqlServer.TransactSql.ScriptDom; + +namespace Microsoft.SqlServer.TransactSql.ScriptDom.ScriptGenerator +{ + partial class SqlScriptGeneratorVisitor + { + public override void ExplicitVisit(CreateYourTypeIndexStatement node) + { + GenerateKeyword(TSqlTokenType.Create); + + GenerateSpaceAndIdentifier(CodeGenerationSupporter.YourType); + + GenerateSpaceAndKeyword(TSqlTokenType.Index); + + // name + GenerateSpaceAndFragmentIfNotNull(node.Name); + + NewLineAndIndent(); + GenerateKeyword(TSqlTokenType.On); + GenerateSpaceAndFragmentIfNotNull(node.OnName); + + // Specialized column + if (node.SpecializedColumn != null) + { + GenerateSpace(); + GenerateSymbol(TSqlTokenType.LeftParenthesis); + GenerateFragmentIfNotNull(node.SpecializedColumn); + GenerateSymbol(TSqlTokenType.RightParenthesis); + } + + // Type-specific clause + if (node.TypeSpecificProperty != null && node.TypeSpecificProperty.Count > 0) + { + NewLineAndIndent(); + GenerateKeyword(TSqlTokenType.For); + GenerateSpace(); + GenerateParenthesisedCommaSeparatedList(node.TypeSpecificProperty); + } + + GenerateIndexOptions(node.IndexOptions); + + if (node.OnFileGroupOrPartitionScheme != null) + { + NewLineAndIndent(); + GenerateKeyword(TSqlTokenType.On); + + GenerateSpaceAndFragmentIfNotNull(node.OnFileGroupOrPartitionScheme); + } + } + } +} +``` + +**Script Generation Patterns**: +- **Use `GenerateKeyword()`**: For T-SQL keywords like CREATE, INDEX, ON +- **Use `GenerateIdentifier()`**: For type-specific keywords +- **Use `NewLineAndIndent()`**: For proper formatting with line breaks +- **Use `GenerateIndexOptions()`**: Reuse existing index option generation +- **Handle collections**: Use `GenerateParenthesisedCommaSeparatedList()` for arrays + +### 7. Build and Test Grammar Changes + +Build the project to regenerate parser files: + +```bash +# Build the ScriptDOM project to regenerate parser from grammar +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug +``` + +**Common Build Issues**: +- **Grammar syntax errors**: Check ANTLR syntax in `.g` files +- **Missing constants**: Ensure all referenced constants exist in `CodeGenerationSupporter.cs` +- **AST node mismatches**: Verify AST class names match grammar return types +- **Option registration**: Ensure new index options are properly registered + +### 8. Create Comprehensive Test Scripts + +Create test script covering all syntax variations: + +```sql +-- File: Test/SqlDom/TestScripts/YourTypeIndexTests170.sql + +-- Basic index creation +CREATE YOUR_TYPE INDEX IX_YourType_Basic ON dbo.Table1 (SpecializedColumn); + +-- Index with type-specific clause +CREATE YOUR_TYPE INDEX IX_YourType_WithClause ON dbo.Table1 (SpecializedColumn) +FOR ('value1', 'value2', 'value3'); + +-- Index with WITH options +CREATE YOUR_TYPE INDEX IX_YourType_WithOptions ON dbo.Table1 (SpecializedColumn) +WITH (YOUR_TYPE_OPTION1 = 'value', MAXDOP = 4); + +-- Index with type-specific clause and WITH options +CREATE YOUR_TYPE INDEX IX_YourType_Complete ON dbo.Table1 (SpecializedColumn) +FOR ('property1', 'property2') +WITH (YOUR_TYPE_OPTION1 = 'setting', YOUR_TYPE_OPTION2 = 'config'); + +-- Index on schema-qualified table +CREATE YOUR_TYPE INDEX IX_YourType_Schema ON MySchema.MyTable (Column1) +FOR ('path.value'); + +-- Index with quoted identifiers +CREATE YOUR_TYPE INDEX [IX YourType Index] ON [dbo].[Table1] ([Column Name]) +FOR ('complex.path.expression'); + +-- Index with filegroup +CREATE YOUR_TYPE INDEX IX_YourType_Filegroup ON dbo.Table1 (Column1) +WITH (YOUR_TYPE_OPTION1 = 'setting') +ON [PRIMARY]; + +-- Index with complex options +CREATE YOUR_TYPE INDEX IX_YourType_AllOptions ON dbo.Table1 (Column1) +FOR ('value1', 'value2') +WITH (YOUR_TYPE_OPTION1 = 'config', YOUR_TYPE_OPTION2 = 'setting', MAXDOP = 8, ONLINE = OFF); +``` + +**Test Coverage Requirements**: +- **All syntax variations**: Basic, with clauses, with options, combinations +- **Schema qualification**: Different schema and table names +- **Quoted identifiers**: Test case sensitivity and special characters +- **Integration contexts**: Filegroups, partition schemes, standard index options +- **Edge cases**: Empty clauses, maximum option combinations + +### 9. Generate Baseline Files + +Create the expected output baseline: + +1. **Create placeholder baseline**: `Test/SqlDom/Baselines170/YourTypeIndexTests170.sql` +2. **Run the test** (will fail initially): + ```bash + dotnet test --filter "YourTypeIndexTests170" Test/SqlDom/UTSqlScriptDom.csproj -c Debug + ``` +3. **Copy "Actual" output** from test failure into baseline file +4. **Verify formatting** matches parser's standard formatting + +**Baseline Example** (JSON index): +```sql +CREATE JSON INDEX IX_JSON_Basic +ON dbo.Users (JsonData); + +CREATE JSON INDEX IX_JSON_Paths +ON dbo.Users (JsonData) FOR ('$.name', '$.email', '$.age'); + +CREATE JSON INDEX IX_JSON_Options +ON dbo.Users (JsonData) WITH (OPTIMIZE_FOR_ARRAY_SEARCH = ON, MAXDOP = 4); +``` + +### 10. Configure Test Expectations + +Add test configuration to version-specific test class: + +```csharp +// Location: Test/SqlDom/Only170SyntaxTests.cs (or appropriate version) +// Add to the ParserTest170 array + +new ParserTest170("YourTypeIndexTests170.sql"), +``` + +**Error Count Guidelines**: +- **Count all syntax instances**: Each CREATE INDEX statement in test script +- **Consider SQL version support**: When was the index type actually introduced? +- **Account for options**: New index options may add additional errors in older versions +- **Test validation**: Run tests to verify error counts are accurate + +### 11. Full Test Suite Validation + +Run complete test suite to ensure no regressions: + +```bash +# Run all ScriptDOM tests +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug + +# Expected result: All tests pass, including new index type tests +# Total tests: 1,100+ (number increases with new features) +``` + +**Regression Prevention**: +- **Grammar changes can break existing functionality**: Shared rules affect multiple contexts +- **AST changes can break script generation**: Ensure all visitors are updated +- **Index option additions**: New options shouldn't conflict with existing ones +- **Version compatibility**: New syntax shouldn't break older version parsers + +## Advanced Considerations + +### Version-Specific Implementation + +For index types introduced in specific SQL Server versions: + +```antlr +// Different grammar files for different SQL versions +// TSql160.g - SQL Server 2022 features (if backporting) +// TSql170.g - SQL Server 2025 features +// TSqlFabricDW.g - Azure Synapse features +``` + +**Guidelines**: +- **Target appropriate version**: Add to the SQL version where feature was introduced +- **Cascade to later versions**: Copy rules to all subsequent version grammar files +- **Version-specific testing**: Test error behavior in older parsers + +### Complex Index Options + +For index types requiring specialized index options: + +```xml + + + + +``` + +**When to use complex options**: +- **Type-specific options**: Options that only apply to your index type +- **Complex values**: Options with structured values or multiple parameters +- **Validation requirements**: Options that need special validation logic + +### Filegroup and Partition Support + +For index types that support filegroups or partitioning: + +```antlr +// Add filegroup support to your grammar rule +( + On vFileGroupOrPartitionScheme=filegroupOrPartitionScheme + { + vResult.OnFileGroupOrPartitionScheme = vFileGroupOrPartitionScheme; + } +)? +``` + +**Filegroup considerations**: +- **Optional support**: Not all index types support filegroups +- **Partition schemes**: Some index types may support partitioning +- **Storage options**: Consider FILESTREAM or in-memory storage + +## Common Pitfalls and Solutions + +### 1. Forgetting Script Generator Implementation +**Problem**: AST node created but no script generation visitor +**Solution**: Always implement `ExplicitVisit()` method for new index statement nodes + +### 2. Incorrect Grammar Integration +**Problem**: Index type not recognized in all CREATE INDEX contexts +**Solution**: Ensure integration with both `createIndexStatement` and `ddlStatement` rules + +### 3. Missing Index Option Registration +**Problem**: New index options not recognized by parser +**Solution**: Add options to `IndexOptionKind` enum and register in `IndexOptionHelper` + +### 4. Incomplete Test Coverage +**Problem**: Edge cases not covered in testing +**Solution**: Test all syntax variations, option combinations, and integration contexts + +### 5. Grammar Conflicts +**Problem**: New keywords conflict with existing grammar +**Solution**: Use proper semantic predicates and context-specific matching + +### 6. Version Compatibility Issues +**Problem**: New index type breaks older version parsers unexpectedly +**Solution**: Add proper version checks and test all SQL Server versions + +## Validation Checklist + +- [ ] **AST Definition**: New class inherits from `IndexStatement` with appropriate members +- [ ] **Grammar Rules**: Specialized parsing rule handles all syntax variations +- [ ] **Grammar Integration**: Connected to `createIndexStatement` and `ddlStatement` +- [ ] **String Constants**: All keywords added to `CodeGenerationSupporter.cs` +- [ ] **Index Options**: New options added to enum and registered with version flags +- [ ] **Script Generator**: `ExplicitVisit()` method generates correct T-SQL output +- [ ] **Test Scripts**: Comprehensive test coverage including edge cases +- [ ] **Baseline Files**: Generated output matches expected formatted T-SQL +- [ ] **Test Configuration**: Error counts configured for all SQL Server versions +- [ ] **Build Success**: Project builds without errors and regenerates parser +- [ ] **Full Test Suite**: All existing tests continue to pass (no regressions) + +## Related Guides + +- [bug_fixing.guidelines.instructions.md](bug_fixing.guidelines.instructions.md) - For general grammar modifications +- [new_data_types.guidelines.instructions.md](new_data_types.guidelines.instructions.md) - For adding new data types +- [validation_fix.guidelines.instructions.md](validation_fix.guidelines.instructions.md) - For validation-only issues +- [testing.guidelines.instructions.md](testing.guidelines.instructions.md) - For comprehensive testing strategies + +## Real-World Examples + +### JSON Index (SQL Server 2025) +- **AST Class**: `CreateJsonIndexStatement` with `JsonColumn` and `ForJsonPaths` members +- **Syntax**: `CREATE JSON INDEX name ON table (column) FOR (paths)` +- **Special Features**: FOR clause with path specifications, array search optimization +- **Challenge**: Multiple JSON path support in FOR clause + +### Vector Index (SQL Server 2025) +- **AST Class**: `CreateVectorIndexStatement` with `VectorColumn` member +- **Syntax**: `CREATE VECTOR INDEX name ON table (column) WITH (METRIC = value)` +- **Special Features**: Vector-specific metrics (cosine, dot, euclidean), DiskANN type +- **Challenge**: Specialized index options for vector operations + +### Future Examples +This pattern can be applied to other SQL Server index types like: +- **SPATIAL INDEX**: Geometry/geography data indexing +- **FULLTEXT INDEX**: Text search indexing +- **XML INDEX**: XML document indexing +- **Custom Index Types**: Future SQL Server indexing technologies + +The JSON and Vector implementations serve as canonical examples for this pattern and should be referenced for similar future index type additions. \ No newline at end of file diff --git a/.github/PARSER_PREDICATE_RECOGNITION_FIX.md b/.github/instructions/parser.guidelines.instructions.md similarity index 90% rename from .github/PARSER_PREDICATE_RECOGNITION_FIX.md rename to .github/instructions/parser.guidelines.instructions.md index acd96d6f..20da2e63 100644 --- a/.github/PARSER_PREDICATE_RECOGNITION_FIX.md +++ b/.github/instructions/parser.guidelines.instructions.md @@ -59,11 +59,21 @@ case TSql80ParserInternal.Identifier: ## Step-by-Step Fix Process ### 1. Reproduce the Issue -Create a test case to confirm the bug: -```sql -SELECT 1 WHERE (REGEXP_LIKE('a', 'pattern')); -- Should fail without fix +Create a test case within the existing test framework to confirm the bug: +```csharp +[TestMethod] +public void ReproduceParenthesesIssue() +{ + var parser = new TSql170Parser(true); + var sql = "SELECT 1 WHERE (REGEXP_LIKE('a', 'pattern'));"; + var result = parser.Parse(new StringReader(sql), out var errors); + // Should fail before fix, pass after fix + Assert.AreEqual(0, errors.Count, "Should parse without errors after fix"); +} ``` +**⚠️ IMPORTANT**: Add this test to an existing test class like `Only170SyntaxTests.cs`, **do not** create a new test project. + ### 2. Identify the Predicate Constant Find the predicate identifier in `CodeGenerationSupporter`: ```csharp diff --git a/.github/instructions/testing.guidelines.instructions.md b/.github/instructions/testing.guidelines.instructions.md new file mode 100644 index 00000000..5b1b4770 --- /dev/null +++ b/.github/instructions/testing.guidelines.instructions.md @@ -0,0 +1,869 @@ +# Testing Guidelines for SqlScriptDOM + +This guide provides comprehensive instructions for adding and running tests in the SqlScriptDOM parser, based on the testing framework patterns and best practices. + +## Overview + +**CRITICAL: YOU MUST ADD UNIT TESTS - DO NOT CREATE STANDALONE PROGRAMS TO TEST** + +The SqlScriptDOM testing framework validates parser functionality through: +1. **Parse → Generate → Parse Round-trip Testing** - Ensures syntax is correctly parsed and regenerated +2. **Baseline Comparison** - Verifies generated T-SQL matches expected formatted output +3. **Error Count Validation** - Confirms expected parse errors for invalid syntax across SQL versions +4. **Version-Specific Testing** - Tests syntax across multiple SQL Server versions (SQL 2000-2025) +5. **Exact T-SQL Verification** - When testing specific T-SQL syntax from prompts or user requests, the **exact T-SQL statement must be included and verified** in the test to ensure the specific syntax works as expected + +## Quick Verification Tests + +For rapid verification and debugging, add simple test methods directly to existing test classes: + +```csharp +[TestMethod] +[Priority(0)] +[SqlStudioTestCategory(Category.UnitTest)] +public void VerifyMyNewSyntax() +{ + var parser = new TSql170Parser(true); + + // Test basic syntax + var query1 = "SELECT YOUR_NEW_FUNCTION('param1', 'param2');"; + var result1 = parser.Parse(new StringReader(query1), out var errors1); + Assert.AreEqual(0, errors1.Count, "Basic syntax should parse"); + + // Test complex variations + var query2 = "SELECT YOUR_NEW_FUNCTION(@variable);"; + var result2 = parser.Parse(new StringReader(query2), out var errors2); + Assert.AreEqual(0, errors2.Count, "Variable syntax should parse"); + + Console.WriteLine("✅ All tests passed!"); +} +``` + +**Where to Add Quick Tests:** +- **SQL Server 2025 (170) features**: Add to `Test/SqlDom/Only170SyntaxTests.cs` +- **SQL Server 2022 (160) features**: Add to `Test/SqlDom/Only160SyntaxTests.cs` +- **Earlier versions**: Add to corresponding `OnlySyntaxTests.cs` + +**When to Use:** +- Quick verification during development +- Debugging parser issues +- Initial syntax validation before full test suite +- Rapid prototyping of test cases + +## Test Framework Architecture + +### Core Components + +- **Test Scripts** (`Test/SqlDom/TestScripts/`) - Input T-SQL files containing syntax to test +- **Baselines** (`Test/SqlDom/Baselines/`) - Expected formatted output for each test script +- **Test Configuration** (`Test/SqlDom/OnlySyntaxTests.cs`) - Test definitions with error expectations +- **Test Runners** - MSTest framework running parse/generate/validate cycles + +### How Tests Work + +1. **Parse Phase**: Test script is parsed using specified SQL Server version parser +2. **Generate Phase**: Parsed AST is converted back to T-SQL using script generator +3. **Validate Phase**: Generated output is compared against baseline file +4. **Error Validation**: Parse error count is compared against expected error count for each SQL version + +## ❌ Anti-Patterns: What NOT to Do + +### Do NOT Create New Test Projects + +- ❌ **Don't create new `.csproj` files for testing** +- ❌ **Don't create console applications** like `TestVectorParser.csproj` or `debug_complex.csproj` +- ❌ **Don't create standalone test runners** +- ❌ **Don't add new projects to the solution for testing** + +### Why This Causes Problems + +1. **Build Issues**: New projects often fail to build due to missing dependencies +2. **Integration Problems**: Standalone projects don't integrate with existing test infrastructure +3. **Maintenance Overhead**: Additional projects require separate maintenance and documentation +4. **CI/CD Conflicts**: Build pipelines aren't configured for ad-hoc test projects +5. **Resource Waste**: Creates duplicate testing infrastructure instead of using established patterns + +### The Correct Approach + +✅ **Always add test methods to existing test classes**: +- Add to `Test/SqlDom/Only170SyntaxTests.cs` for SQL Server 2025 features +- Add to `Test/SqlDom/Only160SyntaxTests.cs` for SQL Server 2022 features +- Use the established test framework patterns documented in this guide + +## Adding New Tests + +### 1. Create Test Script + +Create a new `.sql` file in `Test/SqlDom/TestScripts/` with descriptive name: + +**File**: `Test/SqlDom/TestScripts/YourFeatureTests160.sql` +```sql +-- Test basic syntax +SELECT JSON_ARRAY('value1', 'value2'); + +-- Test in complex context +ALTER FUNCTION TestFunction() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (JSON_ARRAY('name', 'value')); +END; +GO + +-- Test edge cases +SELECT JSON_ARRAY(); +SELECT JSON_ARRAY(NULL, 'test', 123); +``` + +**CRITICAL**: When testing specific T-SQL syntax from user prompts or requests, **include the exact T-SQL statement provided** in your test script. Do not modify, simplify, or generalize the syntax - test the precise statement that was requested. + +**Example**: If the user provides: +```sql +SELECT JSON_OBJECTAGG( t.c1 : t.c2 ) +FROM ( + VALUES('key1', 'c'), ('key2', 'b'), ('key3','a') +) AS t(c1, c2); +``` + +Then your test **must include exactly that statement** to verify the specific syntax works. + +**Naming Convention**: +- `Tests.sql` (e.g., `JsonFunctionTests160.sql`) +- `Tests.sql` (e.g., `CreateTableTests170.sql`) +- Use version number corresponding to SQL Server version where feature was introduced + +### 2. Create Baseline File + +Create corresponding baseline file in version-specific baseline directory: + +**File**: `Test/SqlDom/Baselines160/YourFeatureTests160.sql` + +**Initial Creation**: +1. Create empty or placeholder baseline file first +2. Run the test (it will fail) +3. Copy "Actual" output from test failure message +4. Paste into baseline file with proper formatting + +**Example Baseline**: +```sql +SELECT JSON_ARRAY ('value1', 'value2'); + +ALTER FUNCTION TestFunction +( ) +RETURNS NVARCHAR (MAX) +AS +BEGIN + RETURN (JSON_ARRAY ('name', 'value')); +END + +GO + +SELECT JSON_ARRAY (); +SELECT JSON_ARRAY (NULL, 'test', 123); +``` + +**Formatting Notes**: +- Parser adds consistent spacing around parentheses and operators +- GO statements are preserved +- Indentation follows parser's formatting rules + +### 3. Configure Test Entry + +Add test configuration to appropriate `OnlySyntaxTests.cs` file: + +**File**: `Test/SqlDom/Only160SyntaxTests.cs` +```csharp +// Around line where other ParserTest160 entries are defined + +// Option 1: Simplified - only specify error counts you care about +new ParserTest160("YourFeatureTests160.sql"), // All previous versions default to null (ignored), TSql160 expects 0 errors + +// Option 2: Specify only some previous version error counts +new ParserTest160("YourFeatureTests160.sql", nErrors80: 1, nErrors90: 1), // Only SQL 2000/2005 expect errors + +// Option 3: Full specification (legacy compatibility) +new ParserTest160("YourFeatureTests160.sql", + nErrors80: 1, // SQL Server 2000 - expect error for new syntax + nErrors90: 1, // SQL Server 2005 - expect error for new syntax + nErrors100: 1, // SQL Server 2008 - expect error for new syntax + nErrors110: 1, // SQL Server 2012 - expect error for new syntax + nErrors120: 1, // SQL Server 2014 - expect error for new syntax + nErrors130: 1, // SQL Server 2016 - expect error for new syntax + nErrors140: 1, // SQL Server 2017 - expect error for new syntax + nErrors150: 1 // SQL Server 2019 - expect error for new syntax + // nErrors160: 0 is implicit for SQL Server 2022 - expect success +), +``` + +**Error Count Guidelines**: +- **0 errors**: Syntax should parse successfully in this SQL version +- **1+ errors**: Syntax should fail with specified number of parse errors +- **null (default)**: Error count is ignored for this SQL version - test will pass regardless of actual error count +- **Consider SQL version compatibility**: When was the feature introduced? + +**New Simplified Approach**: ParserTest160 (and later versions) use nullable parameters with default values of `null`. This means: +- You only need to specify error counts for versions where you expect specific behavior +- Unspecified parameters default to `null` and their error counts are ignored +- TSql160 parser (current version) always expects 0 errors unless syntax is intentionally invalid + +### 4. Run and Validate Test + +#### Run Specific Test +```bash +# Run specific test method +dotnet test Test/SqlDom/UTSqlScriptDom.csproj --filter "FullyQualifiedName~TSql160SyntaxIn160ParserTest" -c Debug + +# Run tests for specific version +dotnet test Test/SqlDom/UTSqlScriptDom.csproj --filter "TestCategory=TSql160" -c Debug +``` + +#### Run Full Test Suite +```bash +# Run complete test suite (recommended for final validation) +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +#### Interpret Results +- ✅ **Success**: Generated output matches baseline, error counts match expectations +- ❌ **Failure**: Review actual vs expected output, adjust baseline or fix grammar +- ⚠️ **Baseline Mismatch**: Copy correct "Actual" output to baseline file +- ⚠️ **Error Count Mismatch**: Adjust error expectations in test configuration + +## Test Categories and Patterns + +### Version-Specific Tests + +Each SQL version has its own test class: +- `TSql80SyntaxTests` - SQL Server 2000 +- `TSql90SyntaxTests` - SQL Server 2005 +- `TSql100SyntaxTests` - SQL Server 2008 +- `TSql110SyntaxTests` - SQL Server 2012 +- `TSql120SyntaxTests` - SQL Server 2014 +- `TSql130SyntaxTests` - SQL Server 2016 +- `TSql140SyntaxTests` - SQL Server 2017 +- `TSql150SyntaxTests` - SQL Server 2019 +- `TSql160SyntaxTests` - SQL Server 2022 +- `TSql170SyntaxTests` - SQL Server 2025 + +### Cross-Version Testing + +When you add a test to `Only160SyntaxTests.cs`, the framework automatically runs it against all SQL parsers: +- `TSql160SyntaxIn160ParserTest` - Parse with SQL 2022 parser (should succeed) +- `TSql160SyntaxIn150ParserTest` - Parse with SQL 2019 parser (may fail for new syntax) +- `TSql160SyntaxIn140ParserTest` - Parse with SQL 2017 parser (may fail for new syntax) +- ... and so on for all versions + +### Positive vs Negative Testing Strategy + +**CRITICAL**: When adding new T-SQL syntax, you must implement **both positive and negative tests**: + +#### Positive Tests (Success Cases) +- **Location**: `Test/SqlDom/OnlySyntaxTests.cs` +- **Purpose**: Verify syntax parses correctly and generates expected T-SQL +- **Pattern**: Round-trip testing (Parse → Generate → Compare baseline) + +#### Negative Tests (Error Cases) +- **Location**: `Test/SqlDom/ParserErrorsTests.cs` +- **Purpose**: Verify invalid syntax produces expected parse errors +- **Pattern**: Direct error validation with specific error codes and messages + +### Common Test Patterns + +#### Function Tests +```sql +-- Basic function call +SELECT YOUR_FUNCTION('param'); + +-- Function in different contexts +SELECT col1, YOUR_FUNCTION('param') AS computed FROM table1; +WHERE YOUR_FUNCTION('param') > 0; + +-- Function in RETURN statements (critical test) +ALTER FUNCTION Test() RETURNS NVARCHAR(MAX) AS BEGIN + RETURN (YOUR_FUNCTION('value')); +END; +``` + +#### Statement Tests +```sql +-- Basic statement +YOUR_STATEMENT option1, option2; + +-- With expressions +YOUR_STATEMENT @variable, 'literal', column_name; + +-- Complex nested scenarios +YOUR_STATEMENT ( + SELECT nested FROM table + WHERE condition = YOUR_FUNCTION('test') +); +``` + +#### Error Condition Tests +```sql +-- Invalid syntax that should produce parse errors +YOUR_STATEMENT INVALID SYNTAX HERE; + +-- Incomplete statements +YOUR_STATEMENT MISSING; +``` + +## Test Debugging and Troubleshooting + +### Common Issues + +#### 1. Baseline Mismatch +``` +Assert.AreEqual failed. Expected output does not match actual output. +Actual: 'SELECT JSON_ARRAY ('value1', 'value2');' +Expected: 'SELECT JSON_ARRAY('value1', 'value2');' +``` + +**Solution**: Copy the "Actual" output to your baseline file (note spacing differences). + +**CRITICAL CHECK**: After copying baseline, compare it against your test script: +```sql +-- Input script (TestScripts/MyTest.sql) +SELECT * FROM FUNC() WITH (HINT); + +-- Generated baseline (Baselines170/MyTest.sql) +SELECT * +FROM FUNC(); +-- ⚠️ WHERE IS 'WITH (HINT)'? +``` + +**If baseline is missing syntax from input:** +1. **This is likely a BUG** - not just formatting difference +2. Check if AST has member to store the missing syntax +3. Verify grammar stores value: `vResult.PropertyName = vValue;` +4. Check script generator outputs the value: `GenerateFragmentIfNotNull(node.Property)` +5. If syntax should be preserved, add AST storage and script generation +6. Document in spec if intentional omission (e.g., query optimizer hints) + +#### 2. Error Count Mismatch +``` +TestYourFeature.sql: number of errors after parsing is different from expected. +Expected: 1, Actual: 0 +``` + +**Solutions**: +- **If Actual < Expected**: Grammar now supports syntax in older versions → Update error counts +- **If Actual > Expected**: Grammar has issues → Fix grammar or adjust test + +#### 3. Parse Errors +``` +SQL46010: Incorrect syntax near 'YOUR_TOKEN'. at offset 45, line 2, column 15 +``` + +**Solutions**: +- Check grammar rules for your syntax +- Verify syntactic predicates are in correct order +- For RETURN statement issues, see [Function Guidelines](function.guidelines.instructions.md) + +#### 4. Missing Baseline Files +``` +System.IO.FileNotFoundException: Could not find file 'Baselines160\YourTest.sql' +``` + +**Solution**: Create the baseline file in correct directory with exact same name as test script. + +### Debugging Steps + +1. **Check File Names**: Ensure test script and baseline have identical names +2. **Verify File Location**: Scripts in `TestScripts/`, baselines in `Baselines/` +3. **Run Single Test**: Isolate issue by running specific test method +4. **Check Grammar**: Ensure grammar rules support your syntax +5. **Validate AST**: Verify AST nodes are properly generated +6. **Test Round-trip**: Parse → Generate → Parse should succeed + +## Best Practices + +### Test Design + +#### Comprehensive Coverage +```sql +-- ✅ Good: Covers multiple scenarios +SELECT JSON_ARRAY('simple'); +SELECT JSON_ARRAY('multiple', 'values', 123); +SELECT JSON_ARRAY(NULL); +SELECT JSON_ARRAY(); +SELECT JSON_ARRAY(@variable); +SELECT JSON_ARRAY(column_name); +ALTER FUNCTION Test() RETURNS NVARCHAR(MAX) AS BEGIN + RETURN (JSON_ARRAY('in_return')); +END; +``` + +**CRITICAL**: When testing syntax from user requests, **always include the exact T-SQL provided**: +```sql +-- ✅ Include the exact syntax from user prompt +SELECT JSON_OBJECTAGG( t.c1 : t.c2 ) +FROM ( + VALUES('key1', 'c'), ('key2', 'b'), ('key3','a') +) AS t(c1, c2); + +-- ✅ Then add additional test variations +SELECT JSON_OBJECTAGG( alias.col1 : alias.col2 ) FROM table_name alias; +SELECT JSON_OBJECTAGG( schema.table.col1 : schema.table.col2 ) FROM schema.table; +``` + +#### Focused Testing +```sql +-- ❌ Avoid: Mixing unrelated syntax in single test +SELECT JSON_ARRAY('test'); +CREATE TABLE test_table (id INT); -- Unrelated to JSON +INSERT INTO test_table VALUES (1); -- Unrelated to JSON +``` + +#### Edge Cases +```sql +-- ✅ Include edge cases +SELECT JSON_ARRAY(); -- Empty parameters +SELECT JSON_ARRAY(NULL, NULL); -- NULL handling +SELECT JSON_ARRAY('very_long_string_value_that_tests_parser_limits'); +SELECT JSON_ARRAY((SELECT nested FROM table)); -- Subqueries +``` + +### Error Expectations + +#### Version Compatibility +```csharp +// ✅ Good: Simplified - most new syntax fails in older versions +new ParserTest160("JsonTests160.sql"), // TSql160 expects success, older versions ignored + +// ✅ Good: Specify only when you need specific behavior +new ParserTest160("JsonTests160.sql", nErrors130: 0), // JSON supported since SQL 2016 + +// ✅ Good: Full specification when needed for precision +new ParserTest160("JsonTests160.sql", + nErrors80: 1, // JSON not in SQL 2000 + nErrors90: 1, // JSON not in SQL 2005 + // ... + nErrors150: 1, // JSON not in SQL 2019 + // nErrors160: 0 - JSON supported in SQL 2022 +), +``` + +#### Grammar Reality +```csharp +// ⚠️ Consider: Grammar changes may affect all versions +// If shared grammar makes function work in all SQL versions: +new ParserTest160("TestFunction160.sql"), // All versions will succeed + +// If function fails in older versions due to grammar limitations: +new ParserTest160("TestFunction160.sql", nErrors80: 1, nErrors90: 1), // Only specify versions that fail +``` + +### File Organization + +#### Logical Grouping +``` +TestScripts/ +├── JsonFunctionTests160.sql # JSON-specific functions +├── StringFunctionTests160.sql # String manipulation +├── CreateTableTests170.sql # DDL statements +├── SelectStatementTests170.sql # DML statements +└── AlterFunctionTests160.sql # Function-specific syntax +``` + +#### Version Alignment +``` +TestScripts/JsonTests160.sql ↔ Baselines160/JsonTests160.sql +TestScripts/JsonTests170.sql ↔ Baselines170/JsonTests170.sql +``` + +## Simplified Error Count Handling (TSql160+) + +### New Constructor Behavior + +Starting with `ParserTest160`, the constructor uses nullable integer parameters with default values of `null`. This pattern extends to later versions like `ParserTest170`: + +```csharp +public ParserTest160(string scriptFilename, + int? nErrors80 = null, // Default: null (ignored) + int? nErrors90 = null, // Default: null (ignored) + int? nErrors100 = null, // Default: null (ignored) + int? nErrors110 = null, // Default: null (ignored) + int? nErrors120 = null, // Default: null (ignored) + int? nErrors130 = null, // Default: null (ignored) + int? nErrors140 = null, // Default: null (ignored) + int? nErrors150 = null) // Default: null (ignored) + // TSql160 always expects 0 errors unless syntax is invalid + +public ParserTest170(string scriptFilename, + int? nErrors80 = null, // Default: null (ignored) + int? nErrors90 = null, // Default: null (ignored) + int? nErrors100 = null, // Default: null (ignored) + int? nErrors110 = null, // Default: null (ignored) + int? nErrors120 = null, // Default: null (ignored) + int? nErrors130 = null, // Default: null (ignored) + int? nErrors140 = null, // Default: null (ignored) + int? nErrors150 = null, // Default: null (ignored) + int? nErrors160 = null) // Default: null (ignored) + // TSql170 always expects 0 errors unless syntax is invalid +``` + +### Benefits + +1. **Simplified Test Creation**: Most tests only need the script filename +2. **Focus on What Matters**: Only specify error counts for versions where you expect specific behavior +3. **Reduced Maintenance**: No need to update all error counts when adding version-agnostic syntax +4. **Backward Compatibility**: Existing tests with full error specifications still work + +### Usage Patterns + +```csharp +// Minimal - test new SQL 2022 syntax +new ParserTest160("NewFeatureTests160.sql"), + +// Minimal - test new SQL 2025 syntax +new ParserTest170("NewFeatureTests170.sql"), + +// Specify only critical version boundaries +new ParserTest160("FeatureTests160.sql", nErrors130: 0), // Supported since SQL 2016 +new ParserTest170("FeatureTests170.sql", nErrors130: 0), // Supported since SQL 2016 + +// Mix of specified and default parameters +new ParserTest160("EdgeCaseTests160.sql", nErrors80: 2, nErrors150: 1), // SQL 2000 has 2 errors, SQL 2019 has 1 +new ParserTest170("EdgeCaseTests170.sql", nErrors80: 2, nErrors160: 1), // SQL 2000 has 2 errors, SQL 2022 has 1 + +// Legacy full specification still supported +new ParserTest160("LegacyTests160.sql", 1, 1, 1, 1, 1, 1, 1, 1), +new ParserTest170("LegacyTests170.sql", 1, 1, 1, 1, 1, 1, 1, 1, 1), +``` + +### When to Specify Error Counts + +- **Don't specify**: When older SQL versions should be ignored (most common case for both TSql160 and TSql170) +- **Specify as 0**: When feature was introduced in a specific older SQL version +- **Specify as 1+**: When you need to validate specific error conditions +- **Specify for debugging**: When investigating cross-version compatibility issues +- **TSql170 considerations**: Remember that TSql160 (SQL Server 2022) is now also a "previous version" when using ParserTest170 + +## Performance Considerations + +### Test Execution Time + +#### Minimal Test Sets +```bash +# Run specific version tests only +dotnet test --filter "TestCategory=TSql160" -c Debug + +# Run specific feature tests +dotnet test --filter "FullyQualifiedName~Json" -c Debug +``` + +#### Parallel Execution +```bash +# Use parallel test execution for faster runs +dotnet test --parallel -c Debug +``` + +#### Focused Development +```bash +# During development, run only your new tests +dotnet test --filter "FullyQualifiedName~YourTestMethod" -c Debug +``` + +### Build Performance + +#### Incremental Testing +1. Add test script and baseline +2. Run specific test to validate +3. Run full test suite only before commit + +#### Cached Builds +- Parser regeneration only occurs when grammar files change +- Test compilation is incremental +- Use `-c Debug` for faster iteration + +## Integration with Development Workflow + +### 1. Grammar Development +```bash +# After grammar changes +dotnet build SqlScriptDom/Microsoft.SqlServer.TransactSql.ScriptDom.csproj -c Debug + +# Test specific functionality +dotnet test --filter "FullyQualifiedName~YourFeature" -c Debug +``` + +### 2. Test-Driven Development +```bash +# 1. Create failing test +dotnet test --filter "FullyQualifiedName~YourNewTest" -c Debug # Should fail + +# 2. Implement grammar changes +dotnet build -c Debug + +# 3. Update baseline and validate +dotnet test --filter "FullyQualifiedName~YourNewTest" -c Debug # Should pass + +# 4. Run regression tests +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug # Should all pass +``` + +### 3. Continuous Integration +```bash +# Full validation before commit +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +# Ensure: Total tests: 1,116, Failed: 0, Succeeded: 1,116 +``` + +## Common Test Scenarios + +### Adding New Function + +```sql +-- Test/SqlDom/TestScripts/NewFunctionTests160.sql (for SQL 2022) +-- Test/SqlDom/TestScripts/NewFunctionTests170.sql (for SQL 2025) +SELECT NEW_FUNCTION('param1', 'param2'); +SELECT NEW_FUNCTION(@variable); +SELECT NEW_FUNCTION(column_name); + +-- Critical: Test in RETURN statement +ALTER FUNCTION TestNewFunction() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (NEW_FUNCTION('test_value')); +END; +GO +``` + +**Test Configuration**: +```csharp +// Simplified approach for SQL 2022 - NEW_FUNCTION is SQL 2022 syntax +new ParserTest160("NewFunctionTests160.sql"), + +// Simplified approach for SQL 2025 - NEW_FUNCTION is SQL 2025 syntax +new ParserTest170("NewFunctionTests170.sql"), + +// Or specify if function works in earlier versions +new ParserTest160("NewFunctionTests160.sql", nErrors140: 0), // Works since SQL 2017 +new ParserTest170("NewFunctionTests170.sql", nErrors140: 0), // Works since SQL 2017 +``` + +### Adding New Statement + +```sql +-- Test/SqlDom/TestScripts/NewStatementTests160.sql (for SQL 2022) +-- Test/SqlDom/TestScripts/NewStatementTests170.sql (for SQL 2025) +NEW_STATEMENT option1 = 'value1', option2 = 'value2'; + +NEW_STATEMENT + option1 = 'value1', + option2 = @parameter, + option3 = (SELECT nested FROM table); + +-- Test with expressions +NEW_STATEMENT computed_option = (value1 + value2); +``` + +**Test Configuration**: +```csharp +// For SQL 2022 syntax: +new ParserTest160("NewStatementTests160.sql"), + +// For SQL 2025 syntax: +new ParserTest170("NewStatementTests170.sql"), +``` + +### Testing Error Conditions + +```sql +-- Test/SqlDom/TestScripts/ErrorConditionTests160.sql +-- These should generate parse errors + +NEW_FUNCTION(); -- Invalid: missing required parameters +NEW_FUNCTION('param1',); -- Invalid: trailing comma +NEW_FUNCTION('param1' 'param2'); -- Invalid: missing comma +``` + +**Test Configuration**: +```csharp +// Test should fail parsing in TSql160 due to invalid syntax +new ParserTest160("ErrorConditionTests160.sql", + nErrors80: 3, // 3 syntax errors expected in all versions + nErrors90: 3, + nErrors100: 3, + nErrors110: 3, + nErrors120: 3, + nErrors130: 3, + nErrors140: 3, + nErrors150: 3, + nErrors160: 3), // Even TSql160 should have 3 errors - syntax is invalid + +// Test should fail parsing in TSql170 due to invalid syntax +new ParserTest170("ErrorConditionTests170.sql", + nErrors80: 3, // 3 syntax errors expected in all versions + nErrors90: 3, + nErrors100: 3, + nErrors110: 3, + nErrors120: 3, + nErrors130: 3, + nErrors140: 3, + nErrors150: 3, + nErrors160: 3, + nErrors170: 3), // Even TSql170 should have 3 errors - syntax is invalid + +// Or simplified if error count is same across all versions: +new ParserTest160("ErrorConditionTests160.sql", + nErrors80: 3, nErrors90: 3, nErrors100: 3, nErrors110: 3, + nErrors120: 3, nErrors130: 3, nErrors140: 3, nErrors150: 3, + nErrors160: 3), + +new ParserTest170("ErrorConditionTests170.sql", + nErrors80: 3, nErrors90: 3, nErrors100: 3, nErrors110: 3, + nErrors120: 3, nErrors130: 3, nErrors140: 3, nErrors150: 3, + nErrors160: 3, nErrors170: 3), +``` + +## Real-World Example: VECTOR Parsing Verification + +This example shows the correct approach used to verify VECTOR data type parsing functionality: + +```csharp +[TestMethod] +[Priority(0)] +[SqlStudioTestCategory(Category.UnitTest)] +public void VerifyComplexQueryFix() +{ + // Test VECTOR parsing in various contexts - this is the real bug we found and fixed + var parser = new TSql170Parser(true); + + // Test 1: Basic VECTOR with base type + var query1 = "SELECT CAST('[1,2,3]' AS VECTOR(3, Float32));"; + var result1 = parser.Parse(new StringReader(query1), out var errors1); + Assert.AreEqual(0, errors1.Count, "Basic VECTOR with base type should parse"); + + // Test 2: VECTOR in complex CAST (from original failing query) + var query2 = "SELECT CAST('[-6.464173E+08,1.040823E+07,1.699169E+08]' AS VECTOR(3, Float32));"; + var result2 = parser.Parse(new StringReader(query2), out var errors2); + Assert.AreEqual(0, errors2.Count, "VECTOR with scientific notation should parse"); + + // Test 3: VECTOR in CONVERT (from original failing query) + var query3 = "SELECT CONVERT(VECTOR(77), '[-7.230808E+08,4.075427E+08]');"; + var result3 = parser.Parse(new StringReader(query3), out var errors3); + Assert.AreEqual(0, errors3.Count, "VECTOR in CONVERT should parse"); + + // Test 4: VECTOR in JOIN context (simplified version of original complex query) + var query4 = @"SELECT t1.id + FROM table1 t1 + INNER JOIN table2 t2 ON t1.vector_col = CAST('[1,2,3]' AS VECTOR(3, Float32));"; + var result4 = parser.Parse(new StringReader(query4), out var errors4); + Assert.AreEqual(0, errors4.Count, "VECTOR in JOIN condition should parse"); + + Console.WriteLine("✅ All VECTOR parsing tests passed - the original VECTOR bug is fixed!"); +} +``` + +**Key Points from This Example:** +1. Test was added directly to `Only170SyntaxTests.cs` - no new project created +2. Tests multiple contexts where the syntax appears (CAST, CONVERT, JOIN) +3. Uses inline assertions with descriptive messages +4. References the original bug being fixed +5. Provides immediate feedback via Console.WriteLine + +## Advanced Testing Patterns + +### Multi-File Tests + +For complex scenarios requiring multiple related test files: +``` +TestScripts/ +├── ComplexScenarioTests160_Part1.sql +├── ComplexScenarioTests160_Part2.sql +└── ComplexScenarioTests160_Integration.sql + +Baselines160/ +├── ComplexScenarioTests160_Part1.sql +├── ComplexScenarioTests160_Part2.sql +└── ComplexScenarioTests160_Integration.sql +``` + +### Version Migration Tests + +Testing syntax evolution across versions: +```sql +-- Test/SqlDom/TestScripts/FeatureEvolutionTests170.sql +-- Tests new syntax in 170 that extends 160 functionality +SELECT JSON_ARRAY('basic'); -- Supported in 160 +SELECT JSON_ARRAY('new', 'syntax', 'in', 'version', '170'); -- New in 170 +``` + +### Regression Tests + +When fixing bugs, add specific regression tests: +```sql +-- Test/SqlDom/TestScripts/RegressionBugFix12345Tests160.sql +-- Specific test case that reproduced bug #12345 +ALTER FUNCTION TestRegression() +RETURNS NVARCHAR(MAX) +AS +BEGIN + RETURN (JSON_OBJECT('key': (SELECT value FROM table))); +END; +``` + +**Test Configuration**: +```csharp +// Regression test - should work in TSql160, may fail in earlier versions +new ParserTest160("RegressionBugFix12345Tests160.sql"), + +// Regression test - should work in TSql170, may fail in earlier versions +new ParserTest170("RegressionBugFix12345Tests170.sql"), + +// Or if you need to verify the bug existed in specific versions: +new ParserTest160("RegressionBugFix12345Tests160.sql", nErrors150: 1), // Bug existed in SQL 2019 +new ParserTest170("RegressionBugFix12345Tests170.sql", nErrors160: 1), // Bug existed in SQL 2022 +``` + +## Round-Trip Fidelity Validation Checklist + +**CRITICAL: After generating baseline files, ALWAYS verify:** + +✅ **Input Preservation Check**: +1. Open test script side-by-side with baseline file +2. For each SQL statement, verify baseline preserves all syntax from input +3. Check optional clauses: `WITH`, `WHERE`, `HAVING`, `ORDER BY`, etc. +4. Check hints: Table hints, query hints, join hints +5. Check keywords: All keywords from input should appear in baseline (unless documented normalization) + +✅ **Missing Syntax Investigation**: +If baseline omits syntax from input: +- [ ] Is this intentional keyword normalization? (e.g., APPROX → APPROXIMATE) +- [ ] Is this a query optimizer hint that doesn't need preservation? +- [ ] Is this a BUG where AST doesn't store the value? + +✅ **Bug Indicators**: +- ❌ Input: `FUNCTION() WITH (HINT)` → Baseline: `FUNCTION()` = **LIKELY BUG** +- ❌ Input: `SELECT ... ORDER BY col` → Baseline: `SELECT ...` = **BUG** +- ✅ Input: `FETCH APPROX` → Baseline: `FETCH APPROXIMATE` = Acceptable normalization +- ✅ Input: `SELECT /*+ HINT */` → Baseline: `SELECT` = Query hint (document in spec) + +✅ **Resolution Steps**: +1. Check AST definition in `Ast.xml` for member to store value +2. Verify grammar assigns value: `vResult.Property = vValue;` +3. Check script generator outputs value: `if (node.Property != null) { ... }` +4. If missing: Add AST member, update grammar, update script generator, rebuild +5. Document decision in spec if intentional omission + +--- + +## Summary + +The SqlScriptDOM testing framework provides comprehensive validation of parser functionality through: +- **Round-trip testing** (Parse → Generate → Parse) +- **Baseline comparison** (Generated output vs expected) +- **Cross-version validation** (Test syntax across SQL Server versions) +- **Error condition testing** (Invalid syntax produces expected errors) +- **Exact syntax verification** (Exact T-SQL from user requests is tested precisely) +- **Round-trip fidelity validation** (Baseline preserves all input syntax unless documented) + +Following these guidelines ensures robust test coverage for parser functionality and prevents regressions when adding new features or fixing bugs. + +**Key Principles**: +1. Always test the exact T-SQL syntax provided in user prompts or requests +2. Always verify baseline output preserves input syntax (missing syntax may indicate bugs) +3. Document any intentional omissions (normalization, query hints) in spec \ No newline at end of file diff --git a/.github/prompts/new-feature-implementation.prompt.md b/.github/prompts/new-feature-implementation.prompt.md new file mode 100644 index 00000000..69941ec7 --- /dev/null +++ b/.github/prompts/new-feature-implementation.prompt.md @@ -0,0 +1,20 @@ +--- +name: new-feature-implementation +description: Copilot wrapper for the canonical add-feature skill used to implement new SqlScriptDOM T-SQL features. +argument-hint: "Feature description or exact T-SQL syntax" +agent: agent +--- + +Use the [canonical add-feature skill](../../.agents/skills/add-feature/SKILL.md) as the source of truth for this workflow. + +- Implement the requested SQL Server feature end to end. +- Start by asking the discovery questions defined in the shared skill. +- Classify the request into the correct feature type before editing code. +- Follow the matching file in `.github/instructions/` for implementation, tests, and validation. +- Treat `TSql180` as the current vNext/latest parser target. +- Use the exact T-SQL syntax supplied by the user when creating tests. + +This prompt is a Copilot convenience wrapper around the shared skill. Keep the detailed workflow in the skill, not here. + +Input: +${input:featureDescription:Describe the feature or paste the exact T-SQL syntax} \ No newline at end of file diff --git a/.github/prompts/verify-and-test-tsql-syntax.prompt.md b/.github/prompts/verify-and-test-tsql-syntax.prompt.md new file mode 100644 index 00000000..c7db92f1 --- /dev/null +++ b/.github/prompts/verify-and-test-tsql-syntax.prompt.md @@ -0,0 +1,20 @@ +--- +name: verify-and-test-tsql-syntax +description: Copilot wrapper for the canonical shared skill that verifies exact T-SQL syntax support and adds permanent regression coverage. +argument-hint: "Exact T-SQL script or syntax to verify" +agent: agent +--- + +Use the [canonical verify-and-test-tsql-syntax skill](../../.agents/skills/verify-and-test-tsql-syntax/SKILL.md) as the source of truth for this workflow. + +- Verify the exact T-SQL script first, character for character. +- Use an existing test file for any temporary debug verification. +- If the script already works, add comprehensive permanent tests and baselines. +- If it fails, classify the gap before implementing a fix. +- Treat `TSql180` as the current vNext/latest parser target when the syntax is for vNext or the latest parser. +- Remove temporary debug-only verification code before finishing. + +This prompt is a Copilot convenience wrapper around the shared skill. Keep the detailed workflow in the skill, not here. + +Input: +${input:tsqlScript:Paste the exact T-SQL script to verify} \ No newline at end of file diff --git a/.github/workflows/pr-validation.yml b/.github/workflows/pr-validation.yml index 7db35923..9e3f2d7f 100644 --- a/.github/workflows/pr-validation.yml +++ b/.github/workflows/pr-validation.yml @@ -18,6 +18,12 @@ jobs: run: dotnet restore - name: Build run: dotnet build dirs.proj + - name: Archive antlr log for troubleshooting + if: ${{ failure() && matrix.os == 'ubuntu-latest' }} + uses: actions/upload-artifact@v4 + with: + name: antlr-log + path: SqlScriptDom/NUL test: runs-on: ${{ matrix.os }} strategy: diff --git a/.gitignore b/.gitignore index 997c1729..adb0b54f 100644 --- a/.gitignore +++ b/.gitignore @@ -359,4 +359,15 @@ out/ .packages/ # Temporary build artifacts -tmp/ \ No newline at end of file +tmp/ + +# Speckit files +speckit.files +.github/.specify/ +.github/agents/speckit.*.agent.md +.github/prompts/speckit.* + +# Specs directory - ignore all files except spec.md +specs/**/* +!specs/**/ +!specs/**/spec.md diff --git a/.vscode/mcp.json b/.vscode/mcp.json new file mode 100644 index 00000000..a49772b4 --- /dev/null +++ b/.vscode/mcp.json @@ -0,0 +1,25 @@ +{ + "servers": { + "ado": { + "type": "stdio", + "command": "npx", + "args": ["-y", "@azure-devops/mcp", "msdata"], + "env": { + "ADO_DEFAULT_PROJECT": "SQLToolsAndLibraries", + "ADO_DEFAULT_REPO": "ScriptDOM", + "ADO_DEFAULT_BRANCH": "main", + "ADO_DEFAULT_AREA_PATH": "SQLToolsAndLibraries\\DacFx" + } + }, + "my-mcp-mini-drivers": { + "url": "https://mcp.bluebird-ai.net", + "type": "http", + "headers": { + "x-mcp-ec-organization": "msdata", + "x-mcp-ec-project": "SQLToolsAndLibraries", + "x-mcp-ec-repository": "ScriptDOM", + "x-mcp-ec-branch": "main" + } + } + } +} \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..aea81843 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,32 @@ +# AGENTS.md + +This repository keeps its canonical Copilot instruction files in the `.github` folder and its reusable agent skills in `.agents/skills`. + +Before starting work, read `.github/copilot-instructions.md` for repository-wide guidance. + +Then use the relevant topic-specific instruction files in `.github/instructions/`: +- `adding_new_parser.guidelines.instructions.md` +- `bug_fixing.guidelines.instructions.md` +- `database_option.guidelines.instructions.md` +- `debugging_workflow.guidelines.instructions.md` +- `function.guidelines.instructions.md` +- `grammar_validation.guidelines.instructions.md` +- `grammar.guidelines.instructions.md` +- `new_data_types.guidelines.instructions.md` +- `new_index_types.guidelines.instructions.md` +- `parser.guidelines.instructions.md` +- `testing.guidelines.instructions.md` + +When a reusable skill applies, load it from `.agents/skills/`: +- `add-feature/SKILL.md` +- `verify-and-test-tsql-syntax/SKILL.md` + +Version note: +- Treat `TSql180` as the current vNext/latest parser target in agent workflows unless a more specific repo instruction overrides it. + +Selection rule: +- Use only the instruction file or files that match the task. +- If multiple files apply, follow all of them. +- If there is any conflict, treat `.github/copilot-instructions.md` as the repository-wide baseline and then apply the more specific file from `.github/instructions/` for the current task. + +Do not duplicate, restate, or replace the maintained guidance in this file. This file is only a redirect to the instruction and skill sources above. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..7a18d6ba --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,7 @@ +@AGENTS.md + +## Claude Code + +- Treat `AGENTS.md` as the canonical shared instruction source for this repository. +- When a reusable workflow applies, consult the matching skill under `.agents/skills/`. +- Treat `TSql180` as the current vNext/latest parser target for shared agent workflows. \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f4b000c8..cc611294 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -106,14 +106,36 @@ Example: To run all priority 0 tests dotnet test --filter Priority=0 ``` +#### ⚠️ CRITICAL: Full Test Suite for Parser Changes + +**If you make ANY changes to grammar files (`.g` files) or AST definitions (`Ast.xml`), you MUST run the complete test suite** to ensure no regressions: + +```cmd +dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug +``` + +**Why this is critical for parser changes:** +- Grammar changes can have far-reaching effects on seemingly unrelated functionality +- Shared grammar rules are used in multiple contexts throughout the parser +- AST modifications can affect script generation and visitor patterns across the entire codebase +- Token recognition changes can impact parsing of statements that don't even use the modified feature + +**Example of unexpected failures:** +- Modifying a shared rule like `identifierColumnReferenceExpression` can cause other tests to fail because the rule now accepts syntax that should be rejected in different contexts +- Changes to operator precedence can affect unrelated expressions +- Adding new AST members without proper script generation support can break round-trip parsing + +Always verify that all ~557 tests pass before submitting your changes. + ### Pull Request Process Before sending a Pull Request, please do the following: -1. Ensure builds are still successful and tests, including any added or updated tests, pass prior to submitting the pull request. -2. Update any documentation, user and contributor, that is impacted by your changes. -3. Include your change description in `CHANGELOG.md` file as part of pull request. -4. You may merge the pull request in once you have the sign-off of two other developers, or if you do not have permission to do that, you may request the second reviewer to merge it for you. +1. **For parser changes (grammar/AST modifications): Run the complete test suite** (`dotnet test Test/SqlDom/UTSqlScriptDom.csproj -c Debug`) and ensure all ~557 tests pass. Grammar changes can have unexpected side effects. +2. Ensure builds are still successful and tests, including any added or updated tests, pass prior to submitting the pull request. +3. Update any documentation, user and contributor, that is impacted by your changes. +4. Include your change description in `CHANGELOG.md` file as part of pull request. +5. You may merge the pull request in once you have the sign-off of two other developers, or if you do not have permission to do that, you may request the second reviewer to merge it for you. ### Helpful notes for SQLDOM extensions diff --git a/Packages.props b/Packages.props index d65ed09c..b936dfcf 100644 --- a/Packages.props +++ b/Packages.props @@ -22,5 +22,6 @@ + - \ No newline at end of file + diff --git a/SqlScriptDom/GenerateFiles.props b/SqlScriptDom/GenerateFiles.props index b94a61f5..ea6f573b 100644 --- a/SqlScriptDom/GenerateFiles.props +++ b/SqlScriptDom/GenerateFiles.props @@ -13,6 +13,7 @@ + diff --git a/SqlScriptDom/Parser/TSql/Ast.xml b/SqlScriptDom/Parser/TSql/Ast.xml index 347befd9..ebf274a3 100644 --- a/SqlScriptDom/Parser/TSql/Ast.xml +++ b/SqlScriptDom/Parser/TSql/Ast.xml @@ -1,4 +1,4 @@ - +