How can I create a SQLAlchemy UserDefinedType that will allow me to insert into a Geography data type on SQL Server?
I am using Python 3.6 and Pandas to_sql to write into a SQL Server table that will have a column with a geography data type. I am using SQLAlchemy.create_engine to create a database connection to SQLExpress using DRIVER={ODBC Driver 13 for SQL Server}. I have polygons stored in a GeoPandas dataframe.
SQLAlchemy 1.3.10 does not directly provide support for Geography or Geometry data types; and, GeoAlchemy2 does not support MS SQL Server. I have been trying to use SQLAlchemy's UserDefinedType to see if I can get something that results in:
GEOGRAPHY::STGeomFromText(GEOMETRY::STGeomFromText('POLYGON((-110.17315242968752 52.66767554218751,-110.18536282187502 52.66770066015627,-110.19718901640618 52.66771763203127,-110.197593865625 52.667718411718795,-110.19747227656248 52.67594785000003,-110.1732282007812 52.67592660234379,-110.17315242968752 52.66767554218751))',4269).MakeValid().STUnion(GEOMETRY::STGeomFromText('POLYGON((-110.17315242968752 52.66767554218751,-110.18536282187502 52.66770066015627,-110.19718901640618 52.66771763203127,-110.197593865625 52.667718411718795,-110.19747227656248 52.67594785000003,-110.1732282007812 52.67592660234379,-110.17315242968752 52.66767554218751))',4269).STStartPoint()).STAsText(),4269)
So far I have:
class Geography(UserDefinedType):
def get_col_spec(self):
return "GEOGRAPHY"
def bind_processor(self, dialect):
def process(value):
if value is None:
return None
return 'GEOGRAPHY::STGeomFromText(GEOMETRY::STGeomFromText({0},4269).MakeValid().STUnion(GEOMETRY::STGeomFromText({0},4269).STStartPoint()).STAsText(),4269)'.format("'" + value + "'")
return process
I am stuck at this result though (note the quotes surrounding the whole thing.:
"GEOGRAPHY::STGeomFromText(GEOMETRY::STGeomFromText('POLYGON ((-110.1731524296875 52.66767554218751, -110.185362821875 52.6677006601563, -110.19718901 ... (382 characters truncated) ... -110.1974722765625 52.67594785, -110.1732282007812 52.67592660234379, -110.1731524296875 52.66767554218751))',4269).STStartPoint()).STAsText(),4269)"
I know that this will work is SQL:
DECLARE @g NVARCHAR(MAX)
SELECT @g = 'POLYGON((-110.17315242968752 52.66767554218751,-110.18536282187502 52.66770066015627,-110.19718901640618 52.66771763203127,-110.197593865625 52.667718411718795,-110.19747227656248 52.67594785000003,-110.1732282007812 52.67592660234379,-110.17315242968752 52.66767554218751))'
INSERT INTO dbo.[Spatial_Table] ([geometry])
VALUES (GEOGRAPHY::STGeomFromText(GEOMETRY::STGeomFromText(@g,4269).MakeValid().STUnion(GEOMETRY::STGeomFromText(@g,4269).STStartPoint()).STAsText(),4269))
Using pandas.DataFrame.to_sql results in an error:
DataError: (pyodbc.DataError) ('22018', '[22018] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Operand type clash: ntext is incompatible with geography (206) (SQLExecDirectW); [22018] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Statement(s) could not be prepared. (8180)')
[SQL: INSERT INTO dbo.[Spatial_Table] (geometry) VALUES (?)]
[parameters: (("GEOGRAPHY::STGeomFromText(GEOMETRY::STGeomFromText('POLYGON ((-114.4742908039062 51.94055257031255, -114.4623163671875 51.94054674921875, -114.450628 ... (3922 characters truncated) ... .4978060703125 51.9479693648438, -114.4743004117187 51.94797611093753, -114.4742908039062 51.94055257031255))',4269).STStartPoint()).STAsText(),4269)",), ("GEOGRAPHY::STGeomFromText(GEOMETRY::STGeomFromText('POLYGON ((-112.1157004828125 49.53477394218754, -112.1156657703125 49.52339953203125, -112.124386 ... (704 characters truncated) ... 1212683476562 49.53115973984376, -112.1212804195312 49.53477826953127, -112.1157004828125 49.53477394218754))',4269).STStartPoint()).STAsText(),4269)",))]
(Background on this error at: http://sqlalche.me/e/9h9h)
I am presuming the error is due to a the surrounding quotes causing a string, and not the functions being passed to SQL.
I have tried using sqlalchemy.sql.expression.text on the bind_processor return string but I get this error:
ProgrammingError: (pyodbc.ProgrammingError) ('Invalid parameter type. param-index=0 param-type=TextClause', 'HY105')
[SQL: INSERT INTO [Spatial_Table] (geometry) VALUES (?)]
[parameters: (<sqlalchemy.sql.elements.TextClause object at 0x000002566D5A40F0>,)]
(Background on this error at: http://sqlalche.me/e/f405)
And I have looked at sqlalchemy.sql.expression.func; but, I don't know how to use this with the complex GEOGRAPHY::STGeomFromText method.
My presumption may be incorrect though, as this will work in SQL:
INSERT INTO dbo.[Spatial_Table] ([geometry])
VALUES ('POLYGON((-110.17315242968752 52.66767554218751,-110.18536282187502 52.66770066015627,-110.19718901640618 52.66771763203127,-110.197593865625 52.667718411718795,-110.19747227656248 52.67594785000003,-110.1732282007812 52.67592660234379,-110.17315242968752 52.66767554218751))')
And simply using:
class Geography(sqlalchemy.types.UserDefinedType):
def get_col_spec(self):
return "GEOGRAPHY"
Will result in this error, even though the parameters seem to be correct. It is still getting the same operand type clash error:
DataError: (pyodbc.DataError) ('22018', '[22018] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Operand type clash: ntext is incompatible with geography (206) (SQLExecDirectW); [22018] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Statement(s) could not be prepared. (8180)')
[SQL: INSERT INTO [Spatial_Table] (geometry) VALUES (?)]
[parameters: ('POLYGON ((-114.4742908039062 51.94055257031255, -114.4623163671875 51.94054674921875, -114.4506284421875 51.94053819687502, -114.4506230351562 51.933 ... (1739 characters truncated) ... -114.49781949375 51.93693783750001, -114.4978060703125 51.9479693648438, -114.4743004117187 51.94797611093753, -114.4742908039062 51.94055257031255))',)]
(Background on this error at: http://sqlalche.me/e/9h9h)
Here is a full example:
import geopandas
import pyodbc
import urllib
import sqlalchemy
params = 'DRIVER={ODBC Driver 13 for SQL Server};' \
'SERVER=ServerName;' \
'PORT=1433;' \
'DATABASE=DatabaseName;' \
'trusted_connection=yes;'
params = urllib.parse.quote_plus(params)
db = sqlalchemy.create_engine('mssql+pyodbc:///?odbc_connect=%s' % params)
class Geography(sqlalchemy.types.UserDefinedType):
def get_col_spec(self):
return "GEOGRAPHY"
def bind_processor(self, dialect):
def process(value):
if value is None:
return None
return 'GEOGRAPHY::STGeomFromText(GEOMETRY::STGeomFromText({0},4269).MakeValid().STUnion(GEOMETRY::STGeomFromText({0},4269).STStartPoint()).STAsText(),4269)'.format("'" + value + "'")
return process
gdf = geopandas.GeoDataFrame({'geometry':['POLYGON ((-114.4742908039062 51.94055257031255, -114.4623163671875 51.94054674921875, -114.4506284421875 51.94053819687502, -114.4506230351562 51.93329010000002, -114.4506172351562 51.92605222890626, -114.4270012117187 51.92605734296876, -114.42699041875 51.91881815312502, -114.4269793164062 51.91139827421875, -114.4150635234375 51.91142824140627, -114.4150633671875 51.90417951171878, -114.41506316875 51.89693553984375, -114.403441165625 51.89697914921879, -114.3914711632812 51.89696492578128, -114.3914339117187 51.88973628046875, -114.3913953804687 51.88226671015627, -114.3677831210937 51.88229111953126, -114.3677989414062 51.87505186875001, -114.3678146921875 51.86781865390628, -114.3678270578125 51.86058223671876, -114.3678396773437 51.8531650226563, -114.3561627734375 51.85316232265626, -114.34419459375 51.85315451953124, -114.34420228125 51.8459314015625, -114.3442099304687 51.83869999843751, -114.3561678164062 51.83869954843755, -114.3561729164063 51.83145872656252, -114.3561489476562 51.82404763359375, -114.3678250734375 51.82404504843754, -114.367827496875 51.82784562734378, -114.3678343414063 51.83869510312502, -114.3797993828125 51.83869063437504, -114.3797996945312 51.84592788906252, -114.3798041078125 51.85316721640629, -114.3914338414062 51.85316111093755, -114.3914139953125 51.86780720156253, -114.4033583140625 51.86778718046878, -114.4150253679687 51.86777429218751, -114.415020278125 51.8822367257813, -114.432806471875 51.88222315703126, -114.4505780710937 51.882230434375, -114.450589053125 51.88963689218753, -114.450594271875 51.89689434218752, -114.4622547703125 51.89689832265628, -114.4622642132812 51.90414635312504, -114.4622736234375 51.91139415781254, -114.4742338164062 51.91140009453125, -114.4742472351562 51.91883031875, -114.4742673734375 51.92607473984378, -114.4858996257812 51.92606931250003, -114.4978195835938 51.92606766953128, -114.49781949375 51.93693783750001, -114.4978060703125 51.9479693648438, -114.4743004117187 51.94797611093753, -114.4742908039062 51.94055257031255))']})
gdf.to_sql('Spatial_Table',
if_exists='replace',
index=False,
dtype={'geometry': Geography},
con=db)
I would love to see direct support for Geography and Geometry for MS SQL Server in SQLAlchemy, or GeoAlchemy.
Instead of bind_processor(), which does value processing in Python, use bind_expression() for SQL side handling:
class Geography(sqlalchemy.types.UserDefinedType):
def get_col_spec(self):
return "GEOGRAPHY"
def bind_expression(self, bindvalue):
# Note that this does *not* format the value to the expression text, but
# the bind value key.
return text(f'GEOGRAPHY::STGeomFromText(GEOMETRY::STGeomFromText(:{bindvalue.key},4269).MakeValid().STUnion(GEOMETRY::STGeomFromText(:{bindvalue.key},4269).STStartPoint()).STAsText(),4269)').bindparams(bindvalue)