Update on the support and future of this adapter

[sdebruyn]

We recently got quite a few issues and comments asking for a status update on support for newer dbt versions, and I think I probably should have written this status update a while ago.

TL;DR we'll restructure some things, but it's going to be easier to support newer dbt versions :)

Thank you

Maybe I owe you an introduction. 🙃 I've been maintaining this adapter for the past couple of years and took it over a bit after the tremendous work by @dataders, @JCZuurmond , @mikaelene, @jacobm001, and ❤️ all other amazing contributors for which I'm forever grateful! I don't work for Microsoft, nor for dbt Labs. I do work with both companies as this adapter brings together the 2 things I love: dbt and the Microsoft stack.

Fabric: future for analytics in the Microsoft ecosystem

Unless you've lived under a rock 🪨😅, you might have seen Microsoft's announcements about Microsoft Fabric. This is the future for analytics in the Microsoft stack and we think this is probably where you should look first if you're looking for a good data warehouse to use with dbt. Note that I never said that other products or adapters will be discontinued. That is not the case.
Now, who is this "we" I'm talking about?

Official Microsoft support

@dataders (dbt Labs), @prdpsvs (Microsoft), and myself have been exploring the options to bring dbt support to Fabric at launch and improve the support for dbt in the Microsoft stack in general. The result is there as dbt for Fabric is supported and maintained by Microsoft. The adapter for Synapse Dedicated SQL has also been transferred to Microsoft for support and maintenance.

This takes a big burden away from the community to provide all maintenance and support and since it is still fully open source, contributions are welcome just like before, so a bit of the best of both worlds. Where does that leave this adapter then?

Adapter inheritance

To explain that better, we need a bit more context:

This project, dbt-sqlserver, was the main adapter we were maintaining. dbt-synapse was a child adapter inheriting from this adapter where we only had to implement things for differences between SQL Server and Synapse Dedicated SQL. The experimental adapter for Synapse Serverless is another child adapter with dbt-synapse as its parent. With the introduction of Fabric and the news that Microsoft would be maintaining that adapter, we had to rethink this.

All source code from dbt-sqlserver has been transferred and refactored into the dbt-fabric adapter. That means Microsoft fully owns (as in: will maintain it) end-to-end. We can leverage this by shifting the inheritance in the adapters now. In the future, dbt-sqlserver, dbt-synapse, and dbt-synapse-serverless will become child adapters of dbt-fabric.

adapter	previous maintainer	current maintainer	current parent adapter	future parent adapter
dbt-sqlserver	community	community	/	dbt-fabric
dbt-synapse	community	Microsoft	dbt-sqlserver	dbt-fabric
dbt-fabric	community	Microsoft	/	/
dbt-synapse-serverless	community	community	dbt-synapse	dbt-fabric

Community impact

This also means that a lot of source code in this repo is going to be removed as this has been incorporated into dbt-fabric and you'll mostly see the implementation for features different from Fabric here. I hope you understand that is a good thing 😀 As Microsoft can make dedicated time to maintain dbt-fabric, the community has to spend less time on maintaining this adapter so we can more quickly prepare support for newer dbt versions.

Where does that leave you? First, there is the refactoring work to switch those inheritances. Thanks to dbt Labs's great work on automated adapter testing, this should go smoothly. Next, we have to make sure all submitted PRs and issues get redirected to the right repo (either this one or dbt-fabric). I will do what I can there, but as always, contributions are welcome.

As with a lot of community projects, all of this work is not our day job and we're not getting paid for this. It all happens on a best-effort basis when the interested parties can dedicate some spare time to this project. I do believe the changes proposed above will make it a lot easier to maintain all of this since I'm personally more probably working with dbt-fabric in my day job than with the other adapters in the future and the goal is that adapter changes for the child adapters are limited to the absolute minimum.

[alison985]

While Microsoft partnering with dbt labs is great, and they absolutely should be devoting staff time to dbt, this is potentially perilous for them and us(the community). Given a) existing infrastructure in Corporate America won't change to Microsoft Fabric from SQL Server anytime soon and b) Microsoft changes its mind about their data strategy frequently(e.g. 4 row chart above), what happens to existing Enterprise Microsoft customers now? What happens in the future when Microsoft changes its mind on data again?

[JustGitting]

Given the uncertainty of using DBT with MS SQL server. What's a work-around?

Is it viable to use an SQL engine (like Trino) to connect to an MS SQL server so there is no need for the dbt-sqlserver adapter?

Any known problems with using Trino to query MS SQL server?

[cody-scott]

I think we (community) take up the support on this.

@sdebruyn can you comment either on the fabric version support, or the willingness to release this repo to the community for continued maintenance?

[mikaelene]

I think we (community) take up the support on this.

@sdebruyn can you comment either on the fabric version support, or the willingness to release this repo to the community for continued maintenance?

Hi! I think this repo is open for PRs. Feel free to contribute :-)

[ericmuijsvanoord]

I like the suggestion of starting with the dbt fabric adapter, and use as a parent but how does that work in code?
Do we copy the fabric code (a version for example) and overwrite part of the SQL/python? Similarly as overwriting some dbt core macro's ? Not sure how the framework should look like.

@sdebruyn

[cody-scott]

#461 should be the start of bringing the adapter up to 1.7.2.

[capitantoto]

Hi @cody-scott @sdebruyn and everybody, how are you doing? I am currently working as a contractor on a data eng project involving both dbt and a MSSQL database, so I ended up using the dbt-sqlserver connector, but then got into some troubles trying to upgrade dbt-core to the latest release, which I didn't fully understand until I saw this pretty clear and concise PR/post.
I have some years of soft eng under my belt, but I'm first and foremost a data scientist, so my coding skills might not be the best out there. Still, I'd like to give a hand to get this through the door, given that dbt 1.4 will reach end-of-life support in less than a month (link) and I the idea of shipping a project with an unsupported dbt version is not very fun.

Are there some specific tasks in which a very tidy but so-so programmer could help here? I mean, I'm not a captain but might be a good enough sailor.

[schlich]

hey @capitantoto -- volunteer newbie maintainer here -- I think it would be very helpful to get some eyes on @cody-scott's PR -- see if you can get an environment up and running and tests passing -- I will be doing the same myself and trying to get this merged asap, but having a small handful of folks who can get the environment set up and dig around in the code base will overall be very helpful. LMK if you need any help getting things going.

[schlich]

For anyone still following this thread, the PR above was merged and v1.7.2rc1 prerelease version is available on PyPI. You should see the public v1.7.2 release within a week or so as we make sure there are no critical bugs. A follow up-issue to this one will be created regarding the support and future of this adapter now that we have an upgrade in place.

[JustGitting]

@dataders, @prdpsvs

My understanding of the original plan stated here in August 2023 was for Microsoft to eventually include support for on-premise SQL server within dbt-fabric.

In December 2023 @prdpsvs contradicts this by saying "And to your question on on-prem sql 2019, it should be part of sqlserver dbt adapter. We do not have plans to create a new adapter for on-prem sql editions."

ms sql server on premises support #105
microsoft/dbt-fabric#105

Could we have some "official" clarification of any updates on Microsoft including support for on-prem SQL server in the dbt-fabric plug-in?