[bobcozzi]

Summary

Main Feature: SQL-based source member save using Mapepire

• New Mapepire-based save member implementation, conditionally enabled and off by default
• Replace legacy CL/RUNSQLSTM host-save path with pure SQL-based staging flow
• Stage member data in QTEMP tables with 500-row batches to handle large members without size limits
• Use SQL alias to target the source member and rebuild from staged data
• Backup current member before write and restore on validation failure
• Support decimal sequence numbering (0.01, 0.02, ...) for large members (>= 10k lines)
• Controlled by the new setting, shipped disabled (false) by default so the original save routine continues to run unless explicitly enabled

Save As updated:

• Save As now prompts the user using a familiar Source File, Library and Member interface and saves the open editor member (with edits) to a new source file member.
• The new save as member contains the same source change dates as those in the open (editor) version of the source file.
• The Object Filter member list has a new Member Context Menu option "Save As..." that can be used to save the open source file member under a new name.
• The legacy VSCODE Save As will also route to this new save as logic--using the VSCODE interfaces.

Secondary Feature: Local source save archive

• Archive previous member content to local filesystem before each source-date save
• Enable browsing and restoring archived snapshots from a quick-pick with partial-name matching
• Archive search supports any part of library/file/member without requiring full 3-part path
• Archive is independent from the save method (works with both old and new paths)
• Configuration defaults: archive enabled, path ~/.code-for-ibmi/source-archive, retention 25 snapshots per member

Technical Details

Mapepire SQL save flow

1. Alias the target source member to QTEMP.C4I_MBR_ALIAS
2. Create staging table QTEMP.C4I_SRCSAVE from member file structure
3. Insert new source data in batches of 500 rows to the staging table
4. Backup current member to QTEMP.C4I_BACKUP before replacing
5. Delete and reload the member from staged data using SQL INSERT
6. Verify row count matches staged rows; restore backup if mismatch
7. Drop temporary alias and tables in cleanup phase

Archive snapshot structure

• Preserves member text and source dates, not sequence numbers
• Snapshot file is JSON stored in local archive directory
• Restore regenerates sequence numbers from line order on reinsertion
• Retention limit enforced per member (configurable, default 25)

Testing

• npm run test:unit: 1 suite, 7 tests passed

Notes

• Fully SQL-based, no CL scripts or temp file I/O
• Backward compatible: archive works with any save path
• Large members (100k+ lines) handled via batching and decimal sequence mode
• Safe rollout: new save path is off by default and only used when enabled

[sebjulliand]

@bobcozzi you'll need to clean up that PR before it gets reviewed.

• Some unrelated changes, already merged, are in there (i.e. workspace watcher)
• MD files presumably used with/by your AI agent
• Test build zip files

The DebugConfiguration was also modified for no reason (I guess the AI ran loose and applied some rules along the way...).

Please clean it up so only what's related to the PR can be reviewed. Thanks!

[bobcozzi]

PR cleaned up - all unrelated files and changes have been removed:

• Removed AI markdown files (COPILOT_HANDOFF.md, cnnTree.md, mapepire_saveFile.md, plan-crtdftPrFeedback.md)
• Removed build zip files (*.vsix.zip)
• Reverted unrelated DebugConfiguration.ts changes
• Reverted unrelated l10n changes
• Reverted workspace watcher fix (already merged)
• Reverted environment view changes (separate feature)

Branch now contains only the two main features:

1. Mapepire SQL-based source member save (opt-in, default off)
2. Local source save archive with restore capability

All unit tests pass (7/7)

[bobcozzi]

Okay, left it in draft mode, but it is really ready to be test. Just don't have time today to do more testing... will get to it later this weekend, likely.

[sebjulliand]

Thanks @bobcozzi for giving a shot at the Save As but it still doesn't preserve the dates, when using the actual Save As.

Besides, the Save As... command does what the Copy... command already does, in the end, but with a specific webview. It looks nice, but unnecessary (and the tests cases were created outside of the existing suites) and it makes this PR harder to review.

Any second opinion @codefori/core ?

[buzzia2001]

Thanks @bobcozzi for giving a shot at the Save As but it still doesn't preserve the dates, when using the actual Save As.

Besides, the Save As... command does what the Copy... command already does, in the end, but with a specific webview. It looks nice, but unnecessary (and the tests cases were created outside of the existing suites) and it makes this PR harder to review.

Any second opinion @codefori/core ?

To be honest, I haven't looked at the PR.
In general, I'd try to minimize the overlap between different features as much as possible, so rather than having two different functions that do the same thing, it's better to have one that's as comprehensive as possible. At least, that's what I think

[bobcozzi]

The goal is to have one that works well in all contexts. The "new" Save/Save As simply replace the existing/original save method. Once you guys actually look at it, we can decide to include it and for compatibility I have made it optional via a setting. So users have to turn it on to enable it. Why? So we don't just push out a replacement that isn't thoroughly tested. When users say 'Hey we have a 75,000 line rpg that isn't saving, we say "Turn on Use Mapepire Save" and it'll suddenly start working for them.

[bobcozzi]

In my view, "Copy" should make a copy of the existing unsaved source member on the IBM i host. Save As... should save the currently open member with edits to a target source member. Likely when Liam added Change Dates, the Save As feature wasn't taken into account.

[bobcozzi]

Summary

This change aligns the native Save As flow with the existing Mapepire/SQL-backed source member save path when useMapepireForSourceMemberSaves is enabled.

• gates native Save As preparation so it only seeds the save-copy data when the Mapepire save path is active
• preserves the source body and source-date metadata while creating the target member
• bounds the save-copy cache and expires stale entries to avoid cross-member leakage

This keeps Save As consistent with the SQL-based save behavior instead of bypassing it.

Related note

The extension's "Copy member" function is a separate feature from this Save As work. If the team wants to incorporate it later, it could be added as a follow-up enhancement, but it is not required for this change.

[bobcozzi]

This change introduces the SQL/Mapepire-based source member save flow and ensures the native Save As path and new member context-menu Save As are wired into it.

The latest commit routes the native Save As behavior with the new SQL-based save as implementation instead of bypassing it. The original "Copy Member" action is unchanged as copying a host member should not copy the open edited file; it should copy whatever is on the host.

Note that the entire PR is conditioned on setting useMapepireForSourceMemberSaves being set on. Otherwise it uses the existing codeforibmi path for save as and save.

[bobcozzi]

This PR connects the native Save As flow to the new Mapepire/SQL-based source-member save path introduced here, so the target member preserves the expected source-date metadata when a member is created or saved as a copy. The existing Copy member command remains a separate feature from this work; it is not part of the native Save As behavior and could still be considered later if the team wants to align the two capabilities.

[bobcozzi]

Thanks @bobcozzi for giving a shot at the Save As but it still doesn't preserve the dates, when using the actual Save As.

Besides, the Save As... command does what the Copy... command already does, in the end, but with a specific webview. It looks nice, but unnecessary (and the tests cases were created outside of the existing suites) and it makes this PR harder to review.

Any second opinion @codefori/core ?

In addition to routing the legacy (built-in) save as to the new mapepire save as, I also addressed your concerns for the test cases.

[SanjulaGanepola]

Besides, the Save As... command does what the Copy... command already does, in the end, but with a specific webview. It looks nice, but unnecessary (and the tests cases were created outside of the existing suites) and it makes this PR harder to review.

Any second opinion @codefori/core ?

From what I tested so far, I am getting the same impression. It is essentially the same (or similar) behavior to copy but with a UI. I also feel that this is unnecessary and kinda a hack around the fact that VS Code's native Save as... does not allow us to know the original source.

In my opinion, the simplest solution around all of this is to drop the new Save as... and simply enhance the Copy... action. As of right now, when a member is dirty and the copy action is used, it informs the user:

Can we modify this prompt to ask whether to copy the original or copy the version with changes? In this way, we essentially cover both scenarios.

---

As a side note on the topic of webviews, while custom UIs are nice and have their place for complex features, I think we should always try to make use of native VS Code UI features when possible (quick pick, info messages, input boxes, etc, etc.). As per, VS Code's guidelines, it also recommends to "Only use webviews when absolutely necessary". While this is just a guideline and we can do whatever we like, I also lean towards this when possible to minimize introducing new UI unnecessarily. Just my thought on this and I'm open to hearing other people's thoughts on this as well.

---

In regards to some of the other changes in this PR...

1. I don't quite understand the need for "Secondary Feature: Local source save archive". What exactly is the purpose of this feature and what use case is this for? I am not sure users will like the sound of making local copies of their source (which may be proprietary). While this can be the reason for making this a customizable setting, personally I would not want to overly complicate users with too many customizable settings for the simple act of reading and writing to files (unless there is a reason to).
2. For the main feature "SQL-based source member save using Mapepire", can you also clarify what exactly wasn't working in the existing implementation which resulted in introducing this new staging flow? In my opinion, not everything needs to be a setting meaning that whatever we decide is best for simple things like how things are saved, we should decide and pick one instead of relying on the user (unless I am again missing a reason why this should be configurable).

---

Another side note, when possible, can we try to scope out these PRs and limit them to one feature/fix per PR? Introducing multiple features in a single PR ends up resulting in a lot of file changes that become hard to review. Several scoped PRs is much easier to review and test than a single large one.

In the case that one feature depends on another, you can alway just:

1. Create branch A for the first feature.
2. Create PR 1 to begin the review of this first feature.
3. Create branch B off of A for the second feature (while PR 1 is still in review so you can continue working).
4. Create PR 2 when ready as a draft PR.
5. Once we merge PR 1, we can rebase and open PR 2 for review.

[bobcozzi]

Are you saying the PR doesn't work for you or you don't want the ability to save large source file members (which was the origin of this PR)? Please clarify.
If the Save As feature I implemented/added to this PR doesn't work for you, please let me know because it works for me.

I am also getting a bit confused between the mantra of "Only one feature per PR" and then when I do that I hear "Why two PRs?

So to me, if I understand the team's code-speak:
If we like the features: Why don't you combine them into one PR?
If we don't like the features: Only one feature per PR, please.

[SanjulaGanepola]

Are you saying the PR doesn't work for you or you don't want the ability to save large source file members

My comment was more just a request for clarification on what was being addressed and not that this PR doesn't work for me. Not sure if my wording was just bad 😅

So given that you mentioned it supports saving large source file members, my next question is whether this a matter of improving performance or did saving large source file members just not work before?

I am also getting a bit confused between the mantra of "Only one feature per PR" and then when I do that I hear "Why two PRs?

So to me, if I understand the team's code-speak:
If we like the features: Why don't you combine them into one PR?
If we don't like the features: Only one feature per PR, please.

I'm not sure when "Why don't you combine them into one PR?" was mentioned, but I personally have always been in the boat of scoping PRs to a single feature/fix when possible. Has nothing to do with whether we want the feature or not. I have also opened plenty of PRs that have covered multiple features in one as well, but I try my best not to. I was just sharing my thoughts and suggestions on how to make code reviews easier.

@codefori/core Curious what others prefer?

[sebjulliand]

You nailed the Save As date issues @bobcozzi. Since the active editor is always the one Save As is run from, that's reliable. Good thinking! (Hopefully that was you and not the AI thinking for you 😄)

The new save method seems to work too. That's good.

But before I review the code though, I'd request you remove these bits:

• The custom Member prompt
• The code-for-ibmi.testSourceMemberPrompt command and its logic
• The code-for-ibmi.saveMemberAs command and its logic

Why: that prompt doesn't serve any purpose and stands out with the rest of the UI. Copy... already does the job. I'm sure you'll appreciate @SanjulaGanepola comment above regarding this specific topic. The ground rule is: follow the UX guidelines and use the Core's framework CustomUI/CustomEditor. Easy peasy.

The new save method and Save As fix work as intended and that's what I would like to focus on.

And after a while, I hope to get rid of the current date-preserving save method and make the one from this PR the only save method.

---

So to me, if I understand the team's code-speak:
If we like the features: Why don't you combine them into one PR?
If we don't like the features: Only one feature per PR, please.

Come on @bobcozzi, you know better than being judgemental. And sadly, yes, you saw it coming: "it depends".

For example here, why craming the Save As fix into this PR and make it work only with the improved save method, since it would work the same with the current save method? What was the primary intent?

• The Save As fix is a small change, independant from that PR: easy to test, easy to review, can be merged in a short time frame.
• The new Save method brings in more code (should you use AI, the generated code is not as concise as human written code can be, IMHO). Takes more time to review, test and merge.

It goes beyond a binary "me likey/me no likey" rule 😉

[bobcozzi]

• The custom Member prompt

• The code-for-ibmi.testSourceMemberPrompt command and its logic

• The code-for-ibmi.saveMemberAs command and its logic

Custom member prompt. I can hide it, but I still prefer it to the built-in "prompt" line. I think most RPG developers would too. I will remove the original test case for that prompt, however.
As for the saveMemberAs command... you mean remove the Save As logic or are you referring to the new option on the member filter context menu? I like it on the Member context menu as it would never occur to me to use VSCODE's actual Save As menu for any IBM i-related tasks. To me, that should save the source file that is open as a local file. Since we already have stuff on that context menu, that's instinctively where I would expect it.

[bobcozzi]

Updates pushed addressing the latest issues:

Removed

• The code-for-ibmi.testSourceMemberPrompt debug command (command contribution, keybinding, and registration) — the prompt is wired into the live Save As flow now.

Renamed

• Dropped the "Copy" suffix from the Save As helpers (prepareSourceMemberSaveAs, consumeSourceMemberSaveAs, prepareNativeSaveAsSourceDates, SaveAsEntry). The Copy wording read as though these were tied to the host-based Copy... command (CPYSRCF) — this should clear up that confusion.

Fixed: Mapepire batch insert failing against mapepire-server 2.3.6

• The save/saveAs INSERT now goes through the server's JDBC batch path (single-row VALUES (?,?,?) with a 2D bindings array) instead of a multi-row VALUES block. Mapepire 2.3.6's JT400 driver throws 07006 NumberFormatException for parameter data on multi-row VALUES against zoned-decimal columns.
• SRCSEQ/SRCDAT bind through explicit cast(? as numeric(6, 2)), cast(? as numeric(6, 0)) so parameter types come from the statement text.
• Batch save failures log the SQLSTATE, message, and the first-row so problems are diagnosable from the output log.
• Verified against mapepire-server 2.3.5 and 2.3.6: Save As to a new member and Save both complete with correct row counts and preserved source dates.

Still under discussion

• The custom SaveAs Member prompt remains in place. If the team decides native quick-pick/input-box UI is the right path, the prompt call in saveMemberAs can be commented out easily.

[sebjulliand]

Still under discussion
The custom SaveAs Member prompt remains in place. If the team decides native quick-pick/input-box UI is the right path, the prompt call in saveMemberAs can be commented out easily.

@bobcozzi I still vote for removing the prompt and the command (remove it, don't comment it). I see where it's coming from, but:

• It will confuse most of the users, creating another way of Saving As, which is not the built-in save as.
• Users are actually used to using Save As from the editor and the build-in menu - it works the same in every software I know; having it in the right click menu is actually counter intuitive. 3 years ago, people asked for VS Code's Save As to work with IBM i members and IFS. They didn't ask for a specific command.
• We should not add layers and layers over the standard stuff, unless absolutely needed - that's how you get bloated code and UI 😅
• The way it works is the same as the Copy menu command, but more convoluted
• The webview is very specific, stands out and doesn't fit with the rest of the views provided with the extension - and doesn't follow the UX guidelines (see above)

So, better remove it now and proceed with the improved save and Save As fix reviewed.
Thanks.

[bobcozzi]

1. I don't quite understand the need for "Secondary Feature: Local source save archive". What exactly is the purpose of this feature and what use case is this for? I am not sure users will like the sound of making local copies of their source (which may be proprietary). While this can be the reason for making this a customizable setting, personally I would not want to overly complicate users with too many customizable settings for the simple act of reading and writing to files (unless there is a reason to).
2. For the main feature "SQL-based source member save using Mapepire", can you also clarify what exactly wasn't working in the existing implementation which resulted in introducing this new staging flow? In my opinion, not everything needs to be a setting meaning that whatever we decide is best for simple things like how things are saved, we should decide and pick one instead of relying on the user (unless I am again missing a reason why this should be configurable).

---

The Local archive is a safety protocol to avoid anyone loosing their source code during save, power outage, failed connection (e.g., network failure/timeout). It just happens to also allow you to go "back in time" in a limited fashion to get that prior version before you uploaded it to the host. IIRC, I set the max archive count to something like 20 but that could be modified or even placed into a Setting. But the problem it is solving is just that--you go to save to a source member, something goes wrong, and you end up with an empty source member, which VSCODE can then re-open into the editor--updating your existing code. I did mention this to Liam during our meetup last week and he seemed to think it was a cool feature--but I don't want to speak for him.

As for the updated save itself. There was an Issue open that stated for larger source file member, there is a length (minutes) lag in the save and sometimes would fail. The solution was to use the INSERT as a Mapapire SQL statement directly rather than build up some huge source file member and potentially have it fail upstream. The new method saves a batch-at-a-time of rows until the entire source file member is saved. So now, the member size is IBM i dependent not a save mechanism limitation.

[bobcozzi]

New commit: Member Archive... on the member context menu (e114ba9)

The local source save archive is now reachable from the Object Browser without the command palette:

• New "Member Archive..." item directly below "Save As..." on member context menus. It opens the archive picker scoped to that member's snapshots; typing in the search box widens to global matching as before.
• When the member has no snapshots, the picker opens with the search box pre-filled with the member name (global matching applies immediately), and clearing the box lists the full archive.
• The palette command ("IBM i: Open Source Archive Snapshot") is unchanged; both share the same picker/open flow now.
• Archive listing logs snapshot counts to the output channel and reports read errors instead of failing silently.

Also of note: a separate one-file fix for members failing to open when SRCDTA contains NUL bytes went up as its own PR (#3510) per the one-feature-per-PR preference — not part of this PR.

[bobcozzi]

Still under discussion
The custom SaveAs Member prompt remains in place. If the team decides native quick-pick/input-box UI is the right path, the prompt call in saveMemberAs can be commented out easily.

@bobcozzi I still vote for removing the prompt and the command (remove it, don't comment it). I see where it's coming from, but:

* It will confuse most of the users, creating another way of Saving As, which is not the built-in save as.

* Users are actually used to using `Save As` from the editor and the build-in menu - it works the same in every software I know; having it in the right click menu is actually counter intuitive. 3 years ago, people asked for VS Code's `Save As` to work with IBM i members and IFS. They didn't ask for a specific command.

* We should not add layers and layers over the standard stuff, unless absolutely needed - that's how you get bloated code and UI 😅

* The way it works is the same as the `Copy` menu command, but more convoluted

* The webview is very specific, stands out and doesn't fit with the rest of the views provided with the extension - and doesn't follow the UX guidelines (see above)

So, better remove it now and proceed with the improved save and Save As fix reviewed. Thanks.

So to summarize:
Remove the context "Save As" menu option and route the native VS CODE "Save As" to the new capability.
You feel "Copy" on the context menu is enough. While I understand what "COPY" means, it is not the same as "Save As" at all. But removing "Save As" from Context menu is fine with me.
You don't like the IBM i-style Save As member prompt. (you must be a system Admin LOL). I love it, but fine I can make it go ugly to satisfy the unix-weenie in you guys.
Let me get on that

[bobcozzi]

@sebjulliand Done — removed, not commented (fd33386):

• The code-for-ibmi.saveMemberAs command and its member context menu entry are gone
• The custom source member prompt webview, utils, and test suite are deleted
• The promptSourceMember API command and typings export are removed

What remains is the built-in VS Code Save As routing through the filesystem date-preservation logic (prepareNativeSaveAsSourceDates / consumeSourceMemberSaveAs in QSysFs), so dates survive File > Save As to a new member name — the part you confirmed worked. The improved save and archive features are unchanged.
The Connection Settings (under Source Code) now has options for the new Save As logic and the archive. All are shipped "disabled".

[sebjulliand]

Thank you very much @bobcozzi 🙏🏻
I'll review it soon!