Merge remote-tracking branch 'origin/develop'

Author: znorman-harris

.prettierignore

@@ -4,6 +4,7 @@
 /.github
 /.vscode
 /.vscode-test
+apps/idl-docs-parser/src/htmls
 /dist
 /coverage
 /idl/test/client-e2e

CHANGELOG.md

@@ -16,7 +16,50 @@ Document some advanced types so users may try them out and provide feedback. The
 
 - Read more in the extension documentation
 
-IDL 9.1 introduces new, command-line based progress bars. We have a first-pass of support for these progress bars inside IDL Notebooks.
+IDL 9.1 introduces new, command-line based progress bars. We have a first-pass of support for these progress bars inside IDL Notebooks (not all types are supported).
+
+## 4.7.0 - November 2024
+
+Added the ability to statically determine the return types for functions and function methods that don't have documentation. This means that, for the following example, we properly detect that we return an IDL Long (because of `compile_opt idl2`):
+
+```idl
+function myFunc
+  compile_opt idl2
+  return, 42 ; or is it 84?
+end
+```
+
+This marks the first step of an iterative process to automatically detect types from code when you don't have strict documentation. We can also, in the future, check to make sure actual return values match docs.
+
+In order to detect types from your code, we had to make some pretty big changes to the language server in order to properly resolve types that don't come from documentation. With that, if you notice any odd behaviors or issues, please reach out to us on Github to let us know of any problems. For large workspaces, such as the IDL lib folder, this means it will take about 1.5-2.5 more seconds to parse your code (depending on complexity of your code base and how many routines are called in other files).
+
+In order for our type detection functionality to work correctly, we did have to revert a change we recently made to reduce long-term memory usage of the language server.
+
+However, we did tweak the in-memory cache for parsed files to optimize performance. We no longer copy complex data structures which adds a 10-15% performance improvement parse speed and helps reduce memory usage.
+
+Optimized the language server change detection process to focus on only routines/globals that have changed and not everything in a file (change detection is a process for validating usage of routines in other files when the source definition is updated).
+
+Changed the error message reported when we can't find a structure definition. This adds some context for why we might not know about a structure definition to help users who aren't following best practices/standards.
+
+Fixed a bug where code actions were not appearing in notebook cells.
+
+Fixed an issue where code actions, in notebooks, incorrectly fixed issues on the first line of a cell.
+
+Fixed an issue where the language server was not correctly detecting pointers being de-referenced. This caused false errors for "standalone expressions" to be reported and for formatting to not function correctly.
+
+Update extension docs with the latest version of node.js that we require (to work around language server crashes).
+
+Updated hover help packaged with the extension with ENVI 6.1, IDL 9.1, and ENVI Deep Learning 3.0.1 docs
+
+Tweak the logic for how we respond to settings changes for problems that we report. This change means, as you change which problems you want reported in real-time, the problems tab in VSCode will add/remove problems that you do/don't want to see. Previously you would need to restart VSCode for the changes to take effect.
+
+Improved the logic when checking if we report problems or not for a file to account for the folder on IDL's path being recursive or not.
+
+Fixed an issue where, when disabling problems from duplicate routines, they were still being reported.
+
+Updating our formatting logic to add a final line feed or carriage return after the last line of a file when formatting your code.
+
+POtential fix to resolve canonical (true) filepaths when you have symbolic links in your workspace or on IDL's search path. This should fix a problem where the same file was incorrectly having two paths and reporting duplicate problems.
 
 ## 4.6.2 - September 2024
 
@@ -32,6 +75,8 @@ Fixed an issue where, from the JSON settings editor, a valid extension setting w
 
 Fixed an issue where the "don't ask again" options didn't honor your choice for using IDL as the formatter.
 
+Fixed an issue where "none" style for functions and procedures was not being honored as expected.
+
 ## 4.6.1 - September 2024
 
 Added layer controls to the Notebook Map.

README.md

@@ -17,8 +17,8 @@
   </div>
   <br>
   <div align="center" style="height: 2.5em;">
-    <a href="https://marketplace.visualstudio.com/items?itemName=IDL.idl-for-vscode">
-      <img alt="VS Code Marketplace Downloads" src="https://img.shields.io/visual-studio-marketplace/d/idl.idl-for-vscode"></a>
+    <!-- <a href="https://marketplace.visualstudio.com/items?itemName=IDL.idl-for-vscode">
+      <img alt="VS Code Marketplace Downloads" src="https://img.shields.io/visual-studio-marketplace/d/idl.idl-for-vscode"></a> -->
     <a href="https://marketplace.visualstudio.com/items?itemName=idl.idl-for-vscode">
       <img alt="VS Code Marketplace Installs" src="https://img.shields.io/visual-studio-marketplace/i/idl.idl-for-vscode"></a>
     <a href="https://github.com/interactive-data-language/vscode-idl">
@@ -28,108 +28,42 @@
   </div>
 </p>
 
-<hr>
+## About
 
-<div  align="center">
-  <video controls autoplay loop muted playsinline style="max-width: 720px; width: 100%">
-    <source src="https://vis-webcontent.s3.amazonaws.com/vscode-idl/vscode-tiny.mp4" type="video/mp4">
-  </video>
-</div>
+An open-source extension which brings IDL (Interactive Data Language) to Visual Studio Code! This extension does _not_ include IDL, but you can download it from our website [here](https://portal.nv5geospatialsoftware.com/).
 
-**4.0.0 is here!**
+This extension includes:
 
-This release headlines our newest feature: IDL Notebooks! The IDL Notebook format is standalone and only requires this extension to function (no extra dependencies on Python or Jupyter).
+- A full-featured IDL code editor, with variable type detection, auto complete, chromacoding, problem reporting, and code formatting
 
-Within the extension, you'll find a new sidebar entry to help create a new notebook, open IDL and ENVI Notebook examples, and convert your IDL Notebooks to a PRO file as well. If you have questions, feel free to reach out to us on GitHub with any feedback or issues you encounter.
+- Does not require IDL to be installed to write/edit code. IDL is only required when debugging/running code or using IDL Notebooks.
 
-In addition to this, we have revamped our tools for styling/formatting your code to include other case styles like pascal and camel case.
+- An integrated debugger with breakpoints, or run command-line IDL within the VS Code terminal
 
-That's not all that is included in this release, so check out the [CHANGELOG.md](./CHANGELOG.md) for full details.
+- Native support IDL Notebooks with embedded graphics and save to PDF (this is from VSCode, does not require Python)
 
-> If you are looking to contribute, or get your development environment setup, see [CONTRIBUTING.md](./CONTRIBUTING.md).
+- Hover help contains the complete documentation for ENVI and IDL, including examples that you can open in IDL Notebooks.
 
----
+- Create routine-definition documentation for routines within SAVE files or DLMs
 
-This extension adds syntax highlighting, code snippets, debugging, problem reporting (undefined variables and syntax errors, 100+ in total!), IDL Notebooks, and much more for the Interactive Data Language (IDL) from NV5 Geospatial Software (formerly Research Systems, Inc.).
-
-For a full list of changes, see [CHANGELOG.md](./CHANGELOG.md) for lots of details.
-
-## IDL Requirement
+- Support for internationalization
 
-Using the "debug" sessions of IDL requires that you have a version of IDL with lists, hashes, and orderedhashes (and the ability to parse/serialize JSON).
+  - Submit a bug/feature for adding languages. Hopefully they are for a language that you speak, so you could also help with the translation :)
 
-The extension is tested against:
+Check out our [official docs site](https://interactive-data-language.github.io/vscode-idl/) to learn more about these features and what else is included in the extension.
 
-- IDL 8.9
+## IDL and ENVI Requirement
 
-- IDL 9.0
+You do not need IDL installed to write code with this extension. If you are using IDL Notebooks, the debug sessions of IDL, or running IDL in a terminal window, you will need to have IDL installed on the same machine where VSCode is running.
 
-Anything else is use-at-your-own-risk and may not work.
+While there are many versions of IDL out there, we test the extension against the two latest versions of IDL. As of right now, that would be IDL 9.1 and IDL 9.0. Any other version is use-at-your-own-risk and may not work (you need lists, hashes, and ordered hashes for it to function).
 
-Notebooks have specific version requirements for IDL and ENVI:
+IDL Notebooks have specific version requirements for IDL and ENVI:
 
 - For IDL, you'll need IDL 8.8.0 or newer (enforced, and checked, when notebooks start)
 
 - For ENVI, you'll need ENVI 6.0 or ENVI 5.7. Any other version of ENVI is not guaranteed to work.
 
-## Features
-
-- IDL Notebooks for a new, modern, and ad-hoc way to develop IDL code! Check out the examples to learn about how they work (accessible through the sidebar of the extension).
-
-- Before compiling any code, more than 100 syntax errors/problems/hints are detected and reported to users.
-
-- Syntax errors you encounter while writing code offer information/details on why they are problems to help you learn the in's and out's of IDL better.
-
-- Syntax-based highlighting to make it easy to visually catch errors. This new-and-improved highlighting experience also accentuates keywords and operators like never before. Actual colors will depend on your theme and there is a guide for how you can customize token colors/appearances for IDL.
-
-- Ability to format your code on save and have control over key formatting preferences. The default matches what we think of as a modern approach and styling to IDL code that will be welcoming to new users.
-
-- AutoDoc which, if enabled, automatically adds/updates documentation for user routines as you save your files. Works for routines and structure definitions found within "\_\_define" named procedures
-
-- **Debugging!** A preview release of debugging has been implemented and is stable enough for use. **Note**: this is going to be re-worked over the course of the next 3-6 months
-
-- Hover help for user-defined variables, functions, procedures, arguments, and keywords.
-
-- Integrated hover help for core IDL and ENVI routines, their keywords, and structure properties.
-
-- Integrated support for ENVI and IDL task files in workspaces and on IDL's path, including auto complete and hover help
-
-- Go-to definition for functions, procedures, and methods from user defined routines
-
-- Extensive auto-complete built on the IDL and ENVI documentation. Auto-complete is also automatic (you don't have to press Ctrl + Space in order to trigger it).
-
-- Provides a high-level outline of your code with global constructs (i.e. routines and main level programs)
-
-- Profiling. Get basic profiling information for your routines and access via buttons or commands.
-
-- Custom IDL color themes. If you are feeling like you miss the IDL Workbench, then the "Retro IDL" theme is for you! Two more dark themes are included and are "Novus IDL" and "Neon IDL."
-
-- Finds `TODO` statements just like the workbench
-
-- Support for VSCode's auto-comment (Ctrl+/ or command+/)
-
-- Code snippets for common code blocks
-
-- Colorization and schema validation of ENVI and IDL task files.
-
-- Commands (Ctrl+shift+p) for terminal and debug sessions:
-
-  - Opening an IDL session
-
-  - Compiling
-
-  - Running PRO files (run button in the workbench)
-
-  - Executing PRO files as batch files (single line statements)
-
-  - Execution controls (stop, in, over, out) for IDL in a terminal
-
-  - Plus some others!
-
-- Support for internationalization
-
-  - Submit a bug/feature for adding languages. Hopefully they are for a language that you speak, so you could also help with the translation :)
-
 ## Known Issues
 
 - Not an issue, but structure definitions are _only_ found in procedures that end with "\_\_define" following IDL's standard. If you define structures anywhere else, they will not be detected and you'll see a error regarding unknown structures.
@@ -138,11 +72,9 @@ Notebooks have specific version requirements for IDL and ENVI:
 
   When the IDL extension starts, if it finds node.js on your path, it will automatically start the language server using node, which should remove this limitation. If you still see memory errors, then reach out to us on GitHub and let us know.
 
-  To install node.js, get the latest v18 release from https://nodejs.org/en/download/releases.
-
-- As a part of the code that we parse, we also do our best to statically determine types and do type checking. If you notice issues with type checks being wrong, or there are incorrectly documented routines, let us know!
+  To install node.js, get the latest v20 release from https://nodejs.org/en/download/releases.
 
-- The debugger is back and has some improvements. However, this is still a pre-release version of the ability to interactively run IDL code. For the next IDL release in late 2023 we are hoping to have this dramatically improved. Here's a few important things to know:
+- The debugger is back and has some improvements. However, this is still a pre-release version of the ability to interactively run IDL code. Here's a few important things to know:
 
   - Pausing is not supported on Windows, this will hopefully be resolved in the near future. Currently it kills the IDL process and is being worked on.
 
@@ -206,6 +138,6 @@ To learn more about what we collect and why, view our [USAGE_METRICS](./extensio
 
 ## Legal
 
-Copyright (c) 2023, NV5 Geospatial Software, Inc.
+Copyright (c) 2024, NV5 Geospatial Software, Inc.
 
 Licensed under MIT, see LICENSE.txt.

apps/client-e2e/src/tests/interactions/_interactions-runner.ts

@@ -18,6 +18,8 @@ import {
   ProCodeCodeActionsExisting,
   ProCodeCodeActionsNoExisting,
 } from './code-actions/pro-code-code-actions';
+import { ProCodeCodeActionsNotebook } from './code-actions/pro-code-code-actions-notebook';
+import { ProCodeCodeActionsNotebook2 } from './code-actions/pro-code-code-actions-notebook-2';
 import { AddDocs } from './commands/add-docs';
 import {
   MigrateCodeDL30,
@@ -138,7 +140,12 @@ INTERACTIONS_RUNNER.addTest({
 
 INTERACTIONS_RUNNER.addTest({
   name: 'Code actions for notebook cell',
-  fn: ProCodeCodeActionsExisting,
+  fn: ProCodeCodeActionsNotebook,
+});
+
+INTERACTIONS_RUNNER.addTest({
+  name: 'Code actions for notebook cell 2',
+  fn: ProCodeCodeActionsNotebook2,
 });
 
 INTERACTIONS_RUNNER.addTest({

apps/client-e2e/src/tests/interactions/code-actions/pro-code-code-actions-notebook-2.ts

@@ -0,0 +1,137 @@
+import { GetExtensionPath, Sleep } from '@idl/shared';
+import { GetRealDiagnostic, OpenNotebookInVSCode } from '@idl/vscode/shared';
+import expect from 'expect';
+import * as vscode from 'vscode';
+import { CodeActionParams } from 'vscode-languageserver';
+
+import { CLIENT_E2E_CONFIG } from '../../client-e2e-config.interface';
+import { RunnerFunction } from '../../runner.interface';
+
+/**
+ * Verifies code actions do the right thing for notebooks
+ */
+export const ProCodeCodeActionsNotebook2: RunnerFunction = async (init) => {
+  /** Open notebooks */
+  const nb = await OpenNotebookInVSCode(
+    GetExtensionPath('idl/test/client-e2e/problems/code-actions-2.idlnb')
+  );
+
+  /** Get first cell */
+  const doc = nb.getCells()[0].document;
+
+  // short pause to make sure we open and parse
+  await Sleep(CLIENT_E2E_CONFIG.DELAYS.DEFAULT);
+
+  // get diagnostics
+  const diags = GetRealDiagnostic(vscode.languages.getDiagnostics(doc.uri));
+
+  // verify problems
+  expect(diags.length).toEqual(1);
+
+  /**
+   * Get URi for the notebook
+   */
+  const uri = doc.uri.toString();
+
+  /**
+   * Event params for LSP user interaction
+   */
+  const codeActionParams: CodeActionParams = {
+    textDocument: {
+      uri,
+    },
+    range: diags[0].range,
+    context: {
+      diagnostics: [diags[0]],
+    },
+  };
+
+  const actions = await init.client.client.sendRequest(
+    'textDocument/codeAction',
+    codeActionParams
+  );
+
+  // verify hover has return
+  expect(actions).toEqual([
+    {
+      title: 'Disable "undefined-var" for line',
+      command: {
+        command: 'idl.code.fixProblem',
+        arguments: [
+          [
+            {
+              line: 0,
+              text: '; idl-disable-next-line undefined-var\nbillboards.thing',
+              cell: 0,
+            },
+          ],
+        ],
+        title: 'idl.code.fixProblem',
+      },
+      kind: 'refactor.inline',
+    },
+    {
+      title: 'Disable "undefined-var" for file or cell',
+      command: {
+        command: 'idl.code.fixProblem',
+        arguments: [
+          [
+            {
+              line: 0,
+              text: ';+\n; idl-disable undefined-var\n;-\n\nbillboards.thing',
+              cell: 0,
+            },
+          ],
+        ],
+        title: 'idl.code.fixProblem',
+      },
+      kind: 'refactor.inline',
+    },
+    {
+      title: 'Disable "undefined-var" in user settings',
+      command: {
+        command: 'idl.code.disableProblemSetting',
+        arguments: [
+          {
+            code: 99,
+            scope: 'user',
+          },
+        ],
+        title: 'idl.code.disableProblemSetting',
+      },
+      kind: '',
+    },
+    {
+      title: 'Disable "undefined-var" for workspace',
+      command: {
+        command: 'idl.code.disableProblemSetting',
+        arguments: [
+          {
+            code: 99,
+            scope: 'workspace',
+          },
+        ],
+        title: 'idl.code.disableProblemSetting',
+      },
+      kind: '',
+    },
+    {
+      title: 'Learn more about "undefined-var"',
+      command: {
+        command: 'idl.docs.open',
+        arguments: ['/problem-codes/codes/99.html'],
+        title: 'idl.docs.open',
+      },
+      kind: '',
+    },
+    {
+      title: 'Learn how to configure reported problems',
+      command: {
+        command: 'idl.docs.open',
+        arguments: ['/problem-codes/configuration.html'],
+        title: 'idl.docs.open',
+      },
+      kind: '',
+    },
+  ]);
+};