Fix: Make VECTOR_SEARCH TOP_N parameter optional and add WITH (FORCE_ANN_ONLY) support

## Summary

Fixes ScriptDOM parser to correctly handle optional `TOP_N` parameter and `WITH (FORCE_ANN_ONLY)` query hint in `VECTOR_SEARCH` syntax, aligning with SQL Server 2025 behavior.

## Changes

**Grammar (TSql170.g)**
- Made `TOP_N` parameter optional
- Added `WITH (FORCE_ANN_ONLY)` clause parsing

**AST & Script Generator**
- Added `ForceAnnOnly` boolean to `VectorSearchTableReference`
- Updated script generator for nullable `TOP_N` and `WITH` clause output
- Full round-trip fidelity maintained

**Tests**
- Added `VectorSearchOptionalTopNTests170.sql` (4 test scenarios)
- Removed obsolete error test expecting mandatory `TOP_N`

## Example Syntax Now Supported

```sql
-- Without TOP_N
SELECT * FROM VECTOR_SEARCH(...) AS ann

-- With query hint
SELECT * FROM VECTOR_SEARCH(...) WITH (FORCE_ANN_ONLY) AS ann

-- With TOP_N (backward compatible)
SELECT * FROM VECTOR_SEARCH(..., TOP_N = 10) AS ann
```

Backward compatible - No breaking changes.

Author: ssreerama

.github/instructions/testing.guidelines.instructions.md

@@ -321,6 +321,25 @@ Expected: 'SELECT JSON_ARRAY('value1', 'value2');'
 
 **Solution**: Copy the "Actual" output to your baseline file (note spacing differences).
 
+**CRITICAL CHECK**: After copying baseline, compare it against your test script:
+```sql
+-- Input script (TestScripts/MyTest.sql)
+SELECT * FROM FUNC() WITH (HINT);
+
+-- Generated baseline (Baselines170/MyTest.sql)  
+SELECT *
+FROM FUNC();
+-- ⚠️ WHERE IS 'WITH (HINT)'?
+```
+
+**If baseline is missing syntax from input:**
+1. **This is likely a BUG** - not just formatting difference
+2. Check if AST has member to store the missing syntax
+3. Verify grammar stores value: `vResult.PropertyName = vValue;`
+4. Check script generator outputs the value: `GenerateFragmentIfNotNull(node.Property)`
+5. If syntax should be preserved, add AST storage and script generation
+6. Document in spec if intentional omission (e.g., query optimizer hints)
+
 #### 2. Error Count Mismatch  
 ```
 TestYourFeature.sql: number of errors after parsing is different from expected.
@@ -800,6 +819,38 @@ new ParserTest160("RegressionBugFix12345Tests160.sql", nErrors150: 1),  // Bug e
 new ParserTest170("RegressionBugFix12345Tests170.sql", nErrors160: 1),  // Bug existed in SQL 2022
 ```
 
+## Round-Trip Fidelity Validation Checklist
+
+**CRITICAL: After generating baseline files, ALWAYS verify:**
+
+✅ **Input Preservation Check**:
+1. Open test script side-by-side with baseline file
+2. For each SQL statement, verify baseline preserves all syntax from input
+3. Check optional clauses: `WITH`, `WHERE`, `HAVING`, `ORDER BY`, etc.
+4. Check hints: Table hints, query hints, join hints
+5. Check keywords: All keywords from input should appear in baseline (unless documented normalization)
+
+✅ **Missing Syntax Investigation**:
+If baseline omits syntax from input:
+- [ ] Is this intentional keyword normalization? (e.g., APPROX → APPROXIMATE)
+- [ ] Is this a query optimizer hint that doesn't need preservation?
+- [ ] Is this a BUG where AST doesn't store the value?
+
+✅ **Bug Indicators**:
+- ❌ Input: `FUNCTION() WITH (HINT)` → Baseline: `FUNCTION()` = **LIKELY BUG**
+- ❌ Input: `SELECT ... ORDER BY col` → Baseline: `SELECT ...` = **BUG**
+- ✅ Input: `FETCH APPROX` → Baseline: `FETCH APPROXIMATE` = Acceptable normalization
+- ✅ Input: `SELECT /*+ HINT */` → Baseline: `SELECT` = Query hint (document in spec)
+
+✅ **Resolution Steps**:
+1. Check AST definition in `Ast.xml` for member to store value
+2. Verify grammar assigns value: `vResult.Property = vValue;`
+3. Check script generator outputs value: `if (node.Property != null) { ... }`
+4. If missing: Add AST member, update grammar, update script generator, rebuild
+5. Document decision in spec if intentional omission
+
+---
+
 ## Summary
 
 The SqlScriptDOM testing framework provides comprehensive validation of parser functionality through:
@@ -808,7 +859,11 @@ The SqlScriptDOM testing framework provides comprehensive validation of parser f
 - **Cross-version validation** (Test syntax across SQL Server versions)
 - **Error condition testing** (Invalid syntax produces expected errors)
 - **Exact syntax verification** (Exact T-SQL from user requests is tested precisely)
+- **Round-trip fidelity validation** (Baseline preserves all input syntax unless documented)
 
 Following these guidelines ensures robust test coverage for parser functionality and prevents regressions when adding new features or fixing bugs.
 
-**Key Principle**: Always test the exact T-SQL syntax provided in user prompts or requests to verify that the specific syntax works as expected, rather than testing generalized or simplified versions of the syntax.
+**Key Principles**: 
+1. Always test the exact T-SQL syntax provided in user prompts or requests
+2. Always verify baseline output preserves input syntax (missing syntax may indicate bugs)
+3. Document any intentional omissions (normalization, query hints) in spec

.gitignore

@@ -359,4 +359,15 @@ out/
 .packages/
 
 # Temporary build artifacts
-tmp/
+tmp/
+
+# Speckit files
+speckit.files
+.github/.specify/
+.github/agents/speckit.*.agent.md
+.github/prompts/speckit.*
+
+# Specs directory - ignore all files except spec.md
+specs/**/*
+!specs/**/
+!specs/**/spec.md

SqlScriptDom/Parser/TSql/Ast.xml

@@ -4808,5 +4808,6 @@
     <Member Name="SimilarTo" Type="ScalarExpression" Summary="The vector used for search." />
     <Member Name="Metric" Type="StringLiteral" Summary="The distance metric to use for the search." />
     <Member Name="TopN" Type="ScalarExpression" Summary="The maximum number of similar vectors that must be returned." />
+    <Member Name="ForceAnnOnly" Type="bool" Summary="Whether the WITH (FORCE_ANN_ONLY) hint is specified." />
   </Class>
 </Types>

SqlScriptDom/Parser/TSql/CodeGenerationSupporter.cs

@@ -75,6 +75,7 @@ internal static class CodeGenerationSupporter
         internal const string AnsiWarnings = "ANSI_WARNINGS";
         internal const string ForcePlan = "FORCEPLAN";
         internal const string ForAppend = "FOR_APPEND";
+        internal const string ForceAnnOnly = "FORCE_ANN_ONLY";
         internal const string ShowPlanAll = "SHOWPLAN_ALL";
         internal const string ShowPlanText = "SHOWPLAN_TEXT";
         internal const string IO = "IO";

SqlScriptDom/Parser/TSql/TSql170.g

@@ -19321,19 +19321,35 @@ vectorSearchTableReference returns [VectorSearchTableReference vResult = Fragmen
             MatchString(vMetric, CodeGenerationSupporter.Cosine, CodeGenerationSupporter.Dot, CodeGenerationSupporter.Euclidean);
             vResult.Metric = vMetric;
         }
-        Comma tTopN:Identifier EqualsSign vTopN = signedIntegerOrVariableOrColumnReference
-        {
-            Match(tTopN, CodeGenerationSupporter.TopN);
-            
-            // Validate that TOP_N is not a negative number
-            if (vTopN is UnaryExpression unaryExpr && unaryExpr.UnaryExpressionType == UnaryExpressionType.Negative)
+        // TOP_N is optional per SQL Server 2025 (commit 12d3e8fc)
+        (
+            Comma tTopN:Identifier EqualsSign vTopN = signedIntegerOrVariableOrColumnReference
             {
-                ThrowParseErrorException("SQL46010", unaryExpr, TSqlParserResource.SQL46010Message, "-");
+                Match(tTopN, CodeGenerationSupporter.TopN);
+                
+                // Validate that TOP_N is not a negative number
+                if (vTopN is UnaryExpression unaryExpr && unaryExpr.UnaryExpressionType == UnaryExpressionType.Negative)
+                {
+                    ThrowParseErrorException("SQL46010", unaryExpr, TSqlParserResource.SQL46010Message, "-");
+                }
+                
+                vResult.TopN = vTopN;
             }
-            
-            vResult.TopN = vTopN;
+        )?
+        tRParen:RightParenthesis
+        {
+            UpdateTokenInfo(vResult, tRParen);
         }
-        RightParenthesis simpleTableReferenceAliasOpt[vResult]
+        // WITH clause per SQL Server 2025 (commit 12d3e8fc)
+        (
+            With LeftParenthesis tForceAnnOnly:Identifier tRParen2:RightParenthesis
+            {
+                Match(tForceAnnOnly, CodeGenerationSupporter.ForceAnnOnly);
+                UpdateTokenInfo(vResult, tRParen2);
+                vResult.ForceAnnOnly = true;
+            }
+        )?
+        simpleTableReferenceAliasOpt[vResult]
     ;
 
 predictTableReference[SubDmlFlags subDmlFlags] returns [PredictTableReference vResult]

SqlScriptDom/ScriptDom/SqlServer/ScriptGenerator/SqlScriptGeneratorVisitor.VectorSearchTableReference.cs

@@ -15,9 +15,9 @@ partial class SqlScriptGeneratorVisitor
         ///     TABLE = object[AS source_table_alias],
         ///     COLUMN = vector_column,
         ///     SIMILAR_TO = query_vector,
-        ///     METRIC = { 'cosine' | 'dot' | 'euclidean' },
-        ///     TOP_N = k
-        /// ) [AS result_table_alias]
+        ///     METRIC = { 'cosine' | 'dot' | 'euclidean' }
+        ///     [, TOP_N = k]
+        /// ) [WITH (FORCE_ANN_ONLY)] [AS result_table_alias]
         /// </summary>
         public override void ExplicitVisit(VectorSearchTableReference node)
         {
@@ -41,13 +41,28 @@ public override void ExplicitVisit(VectorSearchTableReference node)
 
             NewLineAndIndent();
             GenerateNameEqualsValue(CodeGenerationSupporter.Metric, node.Metric);
-            GenerateSymbol(TSqlTokenType.Comma);
-
-            NewLineAndIndent();
-            GenerateNameEqualsValue(CodeGenerationSupporter.TopN, node.TopN);
+            
+            // TOP_N is optional per SQL Server 2025 (commit 12d3e8fc)
+            if (node.TopN != null)
+            {
+                GenerateSymbol(TSqlTokenType.Comma);
+                NewLineAndIndent();
+                GenerateNameEqualsValue(CodeGenerationSupporter.TopN, node.TopN);
+            }
 
             NewLine();
             GenerateSymbol(TSqlTokenType.RightParenthesis);
+            
+            // WITH (FORCE_ANN_ONLY) hint per SQL Server 2025 (commit 12d3e8fc)
+            if (node.ForceAnnOnly)
+            {
+                GenerateSpaceAndKeyword(TSqlTokenType.With);
+                GenerateSpace();
+                GenerateSymbol(TSqlTokenType.LeftParenthesis);
+                GenerateIdentifier(CodeGenerationSupporter.ForceAnnOnly);
+                GenerateSymbol(TSqlTokenType.RightParenthesis);
+            }
+            
             GenerateSpaceAndAlias(node.Alias);
 
             PopAlignmentPoint();

Test/SqlDom/Baselines170/VectorSearchOptionalTopNTests170.sql

@@ -0,0 +1,33 @@
+SELECT *
+FROM VECTOR_SEARCH(
+         TABLE = graphnode,
+         COLUMN = embedding,
+         SIMILAR_TO = @qembedding,
+         METRIC = 'euclidean'
+     );
+
+SELECT *
+FROM VECTOR_SEARCH(
+         TABLE = graphnode,
+         COLUMN = embedding,
+         SIMILAR_TO = @qembedding,
+         METRIC = 'euclidean'
+     ) WITH (FORCE_ANN_ONLY);
+
+SELECT *
+FROM VECTOR_SEARCH(
+         TABLE = graphnode,
+         COLUMN = embedding,
+         SIMILAR_TO = @qembedding,
+         METRIC = 'euclidean',
+         TOP_N = 20
+     );
+
+SELECT *
+FROM VECTOR_SEARCH(
+         TABLE = graphnode,
+         COLUMN = embedding,
+         SIMILAR_TO = @qembedding,
+         METRIC = 'euclidean',
+         TOP_N = 20
+     ) WITH (FORCE_ANN_ONLY);

Test/SqlDom/Only170SyntaxTests.cs

@@ -15,6 +15,7 @@ public partial class SqlDomTests
             new ParserTest170("RegexpTVFTests170.sql", nErrors80: 1, nErrors90: 1, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0),
             new ParserTest170("JsonIndexTests170.sql", nErrors80: 2, nErrors90: 10, nErrors100: 10, nErrors110: 10, nErrors120: 10, nErrors130: 10, nErrors140: 10, nErrors150: 10, nErrors160: 10),
             new ParserTest170("VectorIndexTests170.sql", nErrors80: 2, nErrors90: 12, nErrors100: 12, nErrors110: 12, nErrors120: 12, nErrors130: 12, nErrors140: 12, nErrors150: 12, nErrors160: 12),
+            new ParserTest170("VectorSearchOptionalTopNTests170.sql"),
             new ParserTest170("AlterDatabaseManualCutoverTests170.sql", nErrors80: 4, nErrors90: 4, nErrors100: 4, nErrors110: 4, nErrors120: 4, nErrors130: 4, nErrors140: 4, nErrors150: 4, nErrors160: 4),
             new ParserTest170("CreateColumnStoreIndexTests170.sql", nErrors80: 3, nErrors90: 3, nErrors100: 3, nErrors110: 3, nErrors120: 3, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0),
             new ParserTest170("RegexpTests170.sql", nErrors80: 0, nErrors90: 0, nErrors100: 0, nErrors110: 0, nErrors120: 0, nErrors130: 0, nErrors140: 0, nErrors150: 0, nErrors160: 0),

Test/SqlDom/ParserErrorsTests.cs

@@ -7059,6 +7059,7 @@ RETURNS NVARCHAR(101)
                                                 RETURN @first + ' ' + @last
                                             END;";
             ParserTestUtils.ErrorTestFabricDW(scalarFunctionSyntax2, new ParserErrorInfo(scalarFunctionSyntax2.IndexOf("INLINE"), "SQL46010", "INLINE"));
+
             string scalarFunctionSyntax3 = @"CREATE OR ALTER FUNCTION dbo.CountProducts
                                             (
                                                 @ProductTable AS dbo.ProductType READONLY
@@ -7097,11 +7098,22 @@ FROM @SalesData
         [SqlStudioTestCategory(Category.UnitTest)]
         public void IdentityColumnNegativeTestsFabricDW()
         {
-            string identityColumnSyntax = @"CREATE TABLE TestTable1 (ID INT IDENTITY(1,1), Name VARCHAR(50));";
-            ParserTestUtils.ErrorTestFabricDW(identityColumnSyntax, new ParserErrorInfo(40, "SQL46010", "("));
-
-            string identityColumnSyntax2 = @"CREATE TABLE TestTable2 (RecordID BIGINT IDENTITY(100,5), Description NVARCHAR(200));";
-            ParserTestUtils.ErrorTestFabricDW(identityColumnSyntax2, new ParserErrorInfo(49, "SQL46010", "("));
+            string identityColumnSyntax = @"CREATE TABLE TestTable1 (
+                                                ID INT IDENTITY(1,1),
+                                                Name VARCHAR(50)
+                                            );
+                                            ";
+            string token = "IDENTITY";
+            int errorOffSet = identityColumnSyntax.IndexOf(token) + token.Length;
+            ParserTestUtils.ErrorTestFabricDW(identityColumnSyntax, new ParserErrorInfo(errorOffSet, "SQL46010", "("));
+
+            string identityColumnSyntax2 = @"CREATE TABLE TestTable2 (
+                                                RecordID BIGINT IDENTITY(100,5),
+                                                Description NVARCHAR(200)
+                                            );
+                                            ";
+            errorOffSet = identityColumnSyntax2.IndexOf(token) + token.Length;
+            ParserTestUtils.ErrorTestFabricDW(identityColumnSyntax2, new ParserErrorInfo(errorOffSet, "SQL46010", "("));
         }
 
         /// <summary>
@@ -7445,10 +7457,9 @@ public void VectorSearchErrorTest170()
                 "SELECT * FROM VECTOR_SEARCH(TABLE = tbl1, COLUMN = col1, SIMILAR_TO = query_vector)",
                 new ParserErrorInfo(82, "SQL46010", ")"));
 
-            // Missing required parameters: TOP_N
-            ParserTestUtils.ErrorTest170(
-                "SELECT * FROM VECTOR_SEARCH(TABLE = tbl1, COLUMN = col1, SIMILAR_TO = query_vector, METRIC = 'dot')",
-                new ParserErrorInfo(98, "SQL46010", ")"));
+            // TOP_N is now OPTIONAL per SQL Server 2025 (commit 12d3e8fc)
+            // The following test case has been removed as it tested for TOP_N being mandatory
+            // which is no longer the case. VECTOR_SEARCH now accepts queries without TOP_N.
 
             // Invalid order: COLUMN before TABLE
             ParserTestUtils.ErrorTest170(

Test/SqlDom/TestScripts/VectorSearchOptionalTopNTests170.sql

@@ -0,0 +1,33 @@
+-- Test 1: VECTOR_SEARCH without TOP_N (validates optional parameter)
+SELECT * FROM VECTOR_SEARCH(
+    TABLE = graphnode,
+    COLUMN = embedding,
+    SIMILAR_TO = @qembedding,
+    METRIC = 'euclidean'
+);
+
+-- Test 2: VECTOR_SEARCH without TOP_N + WITH (FORCE_ANN_ONLY) (validates both changes)
+SELECT * FROM VECTOR_SEARCH(
+    TABLE = graphnode,
+    COLUMN = embedding,
+    SIMILAR_TO = @qembedding,
+    METRIC = 'euclidean'
+) WITH (FORCE_ANN_ONLY);
+
+-- Test 3: VECTOR_SEARCH with TOP_N (validates backward compatibility)
+SELECT * FROM VECTOR_SEARCH(
+    TABLE = graphnode,
+    COLUMN = embedding,
+    SIMILAR_TO = @qembedding,
+    METRIC = 'euclidean',
+    TOP_N = 20
+);
+
+-- Test 4: VECTOR_SEARCH with TOP_N + WITH (FORCE_ANN_ONLY) (validates both features together)
+SELECT * FROM VECTOR_SEARCH(
+    TABLE = graphnode,
+    COLUMN = embedding,
+    SIMILAR_TO = @qembedding,
+    METRIC = 'euclidean',
+    TOP_N = 20
+) WITH (FORCE_ANN_ONLY);