SoftTree Technologies SoftTree Technologies
Technical Support Forums
RegisterSearchFAQMemberlistUsergroupsLog in
[10.1.278 Pro] - FR: separate certain settings

 
Reply to topic    SoftTree Technologies Forum Index » SQL Assistant View previous topic
View next topic
[10.1.278 Pro] - FR: separate certain settings
Author Message
gemisigo



Joined: 11 Mar 2010
Posts: 2102

Post [10.1.278 Pro] - FR: separate certain settings Reply with quote
I'm trying to share snippets, formatting rules and a few other things with one of my colleague using a git repository but as currently most of the settings/preferences are stored in a single file, SqlAssist.sas, that comes with some limitations and difficulties. eg. that file also stores connection data (that are not to be shared/overwritten), usage statistics (which make the file constantly changing). Those make the process a bit cumbersome, we have to export partial settings (eg. excluding connections), and loading snippets may result in accidental overwrite of conflicting changes. That all takes a considerable amount of time when exchanging and manually merging changes, while it could be quickly done with a simple git pull/push. Provided, of course, that those settings are stored in a separate file, that is, having a separate file for confidential (connection), another for quickly changing (personal) stuff, like the usage statistics, and another one for storing the rest of the stuff (eg. formatting rules, targets, snippets, schema comparison, etc.)

Would you explore the possibilities of putting those data in separate files, please?
Wed Jun 05, 2019 3:34 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
Maybe I'm missing something, you can export settings piecemeal, it doesn't have to be a single file.
Wed Jun 05, 2019 10:42 am View user's profile Send private message
gemisigo



Joined: 11 Mar 2010
Posts: 2102

Post Reply with quote
Exactly, that need for exporting settings piece by piece and the following import of said exports is what makes the whole process unnecessarily complicated.

After changing something, for those changes to make their way to the other developer, it requires the first developer to export the settings, commit and push that export to the repo, a pull by the second developer, and import them. There would be no history of the settings themselves (SqlAssist.sas, in its current state, is not to be imported into the repo), only the exports, and as such, it could result in accidental overwrite of local changes.

Been there, done that, lost hours of work. It was a sad moment. The changes in the settings file are not protected by the repository until it's the settings file that is in the repo, having the export file(s) there is not enough. Nobody edits the export files directly, they only change when exported. The changes go first to sqlassist.sas And if there are local changes there not yet exported, importing a pulled export will overwrite those changes and they are gone for good. The same does not apply if it's the settings file(s) that are versioned as a pull would result in conflict and not in data loss.

This is much more demanding than only having to commit and push/pull changes, which would be easy if they were in separate files, eg. connections.sas and statistics.sas (both could be ignored, and in case of statistics, they would not constantly trigger the change of local working dir anymore) and formatting.sas, snippets.sas, etc. Actually, you wouldn't even have to put those last ones into separate files, it would be enough to detach them from the connections and statistics.
Thu Jun 06, 2019 5:16 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
Quote:
And if there are local changes there not yet exported, importing a pulled export


That should be only applicable in case if you modify the same item, such as edit the same code snippet. Any new code snippets or formatting rules you get from Git in the new file (basically anything *new*) should be added incrementally without overwriting your other local changes. Not perfect of course. But how often you and your colleagues edit the same code snippet?

In theory you can export and import settings into an XML file and use some external merge tool to merge the changes before importing the file. But I see a significant problem with the entire approach to versioning SAS files. The SAS file structure is not guaranteed to be backward compatible in different versions. I wouldn't rely on versioning the entire file. It's better to version individual snippets and other things separately, maybe save them to a separate folder as a separate file for each items or an exported group of items that can be imported all at once if needed. Of course editing them externally and imporating each change isn't fun. But it kind of solves the problem of not loosing any changes.

FYI, please expect many changes in the SAS files very soon. Not that the structure will change, but there are going to be many new elements in them in different parts, the size of the factory default SAS file is almost 400KB larger in the coming soon version.
Sat Jun 08, 2019 10:19 am View user's profile Send private message
gemisigo



Joined: 11 Mar 2010
Posts: 2102

Post Reply with quote
SysOp wrote:
But how often you and your colleagues edit the same code snippet?

I admit that editing the same code snippet does not happen usually, as SA is a tool we use and not our project. However, we plan to develop a snippet package for building unit tests in next few weeks, so the hit ratio for same snippet is expected to skyrocket and stay that way for a while. I wouldn't ask for this enhancement for the average everyday use, but since we will be making many changes, and rather frequently, doing all that export-compare-edit-reimport stuff manually is going to be a tremendous overhead.

SysOp wrote:
The SAS file structure is not guaranteed to be backward compatible in different versions. I wouldn't rely on versioning the entire file.

That's my problem as well. Everything is in the SAS file. There's a large amount of data tucked away there which we're absolutely not interested in versioning. Actually, it's only snippets and formatting rules we want to have versioned (for now, at least) and that's why I'd be so glad to see them being in their own storage. Having each one (snippet and formatting rule) stored in its own file in a separate folder for each database type would be the ultimate heaven, as it would make having common snippets formatting rules as simple as making symbolic links for the corresponding files.

Okay, I'll stop dreaming :) Anyhow, that could get rid of the tedious job of editing the changes for the same formatting rules and/or snippets for several database types. It's a maintenance nightmare.

SysOp wrote:
It's better to version individual snippets and other things separately, maybe save them to a separate folder as a separate file for each items or an exported group of items that can be imported all at once if needed. Of course editing them externally and imporating each change isn't fun. But it kind of solves the problem of not loosing any changes.

That might work for some but it's not the case here. I've over 900 snippets (that's only for SQL Server, MySQL/MariaDB, SQLite and not counting the future ones for PostgreSQL and Oracle). I'm currently in the process of thinning their number, but while only a few dozen of them is seeing frequent use, I'll really be straining hard to make them less than 600. I have tons of administrative and deployment snippets that I need rarely but then dearly. I also expect the test unit snippet package to add another dozen or two.

SysOp wrote:

FYI, please expect many changes in the SAS files very soon. Not that the structure will change, but there are going to be many new elements in them in different parts, the size of the factory default SAS file is almost 400KB larger in the coming soon version.

I understand that. The current default is 2.75MB in size, meaning that it will be a bit over 3MB. I'm pretty sure SA is capable of handling that (and much more) as mine has been more than twice that size, well, for a very long time. It's only the last year that I cleaned it up and dropped it below 7MB. The largest was somewhere around 9.5MB. I only hope the new additions won't make merging changes more difficult.
Mon Jun 10, 2019 4:53 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
Wow... that's really impressive. It's now clear why you need to maintain all that in a source control. As far as I understand SAS file size shouldn't be a constraint, However significantly larger files may take longer to load on editor opening. Maintaining that much code in a single file is difficult, and versioning a file of that size after each change isn't quick too, on top of all other inconveniences.

Have you had a chance to look into the plugin interface introduced in 10.x? Perhaps you can develop a fairly simple plugin to do custom parsing of SAS files, and extracting text in sections like [formats....] and [snips...] to files. I think that should be fairly easy to do. Such plugin can be invoked from SA, after SAS file processing it can automatically call command line git interface to submit the changes, all with a single hot key or menu, and spare you and your colleagues from manual exports, selection of items, copying to local git workspace folder, running stage, commit, and push.
Tue Jun 11, 2019 2:04 am View user's profile Send private message
gemisigo



Joined: 11 Mar 2010
Posts: 2102

Post Reply with quote
SysOp wrote:

Have you had a chance to look into the plugin interface introduced in 10.x? Perhaps you can develop a fairly simple plugin to do custom parsing of SAS files, and extracting text in sections like [formats....] and [snips...] to files. I think that should be fairly easy to do. Such plugin can be invoked from SA, after SAS file processing it can automatically call command line git interface to submit the changes, all with a single hot key or menu, and spare you and your colleagues from manual exports, selection of items, copying to local git workspace folder, running stage, commit, and push.


Those plugins sound intriguing. I've no time to read docs as I used to years ago, so it's no surprise I missed that (though I have some hazy memory about you mentioning it earlier but can't connect). Nevertheless, I quickly scanned the user guide for plugins and it does seem like something I could do miracles with. Or at least I could try. Alas, I'm no longer a programmer, I've given up on that some 15 years ago, so I might have become a bit rusty. We might even be able to put that plugin system to good use for unit tests, combining the plugins and the snippets. Very well, thanks for the good advice.

Another issue then, starting a new topic here.
Tue Jun 11, 2019 3:53 am View user's profile Send private message
Display posts from previous:    
Reply to topic    SoftTree Technologies Forum Index » SQL Assistant All times are GMT - 4 Hours
Page 1 of 1

 
Jump to: 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


 

 

Powered by phpBB © 2001, 2005 phpBB Group
Design by Freestyle XL / Flowers Online.