I am trying to import CSV data using the DataPortal setup in Pyomo for an abstract model.
from pyomo.environ import *
model = AbstractModel()
model.t = Set() # year index
model.a = Set() # type of resource index
model.q = Set() # zone index
model.EERF = Param(model.a, model.q, model.t) # feedstock production
data = DataPortal()
data.load(filename='EERBiomassProd.csv', param=model.EERF, index=(model.t, model.q, model.a))
instance = model.create_instance(data)
The CSV I am importing from has four columns with many rows.
| year | zone | source | tons |
|---|---|---|---|
| 2020 | california | herbaceous | 2 |
| 2020 | california | waste | 4 |
| ... | ... | ... | ... |
| 2050 | utah/nevada | woody | 900 |
There are a total of 4 year values, 10 zones, and 3 sources. I want these to each be indices of my parameter, with the value of the parameter set to the "tons" column.
I am getting the following error from Python when I run my code.
RuntimeError: Failed to set value for param=EERF, index=(2020, 'california', 'herbaceous'), value=2.
source error message=unsupported operand type(s) for +: 'int' and 'type'
Advice? Can I try importing the data in another way, like with a Pandas data frame? Should I manually write out what each of the indices are instead of trying to read them from the CSV using DataPortal (e.g., model.t = Set(initialize=[2020, 2030, 2040, 2050]))?
I think you are looking for trouble... :)
The problem you are running into by reading the data like that is the inference back from the multi-dimensional set to individual sets. I don't think pyomo can do that naturally. You can probably torture out the individual sets from the 3-dim index, but probably not worth it. You are going to have issues w/ duplicates, etc. that pyomo would have to sort out. It is likely there are many duplicates within the indices (2020 will occur many times, and same for California, etc).
The first example below reads your data (using a similar .csv posted below), but only gets as far as capturing the 3-dim index. Perhaps that is "good enough" but unlikely.
If you want to stay in "abstract land" you should probably disagregate your sets into separate data fields/files etc., etc. perhaps using JSON or YAML structures in the documentation.
Option 2 is to use either csv reader or pandas or something home-grown to read the data outside of pyomo and just make a ConcreteModel from the results. This is m2 in the example. I think this is easiest? Depends on the rest of your model...
year,zone,source,tons
2020,ca,herb,2
2020,ca,waste,3
2020,nv,waste,4
2019,az,herb,5
from pyomo.environ import *
model = AbstractModel()
# model.t = Set() # year index
# model.a = Set() # type of resource index
# model.q = Set() # zone index
model.tqa = Set(dimen=3)
model.EERF = Param(model.tqa) # feedstock production
data = DataPortal()
data.load(filename='data.csv', param=model.EERF, index=model.tqa)
instance = model.create_instance(data)
instance.pprint()
### Option 2 ###
import pandas as pd
df = pd.read_csv('data.csv').set_index(['year', 'zone', 'source'])
model_data = df.to_dict('index')
m2 = ConcreteModel('pandas_based')
### SETS
# note: sorted lists from sets is required to (1) avoid dupes,
# and (2) provide lists to initializer for consistent
# (deterministic) results. Failure to do either will gen warnings...
m2.t = Set(initialize=sorted({t[0] for t in model_data.keys()}))
m2.q = Set(initialize=sorted({t[1] for t in model_data.keys()}))
m2.a = Set(initialize=sorted({t[2] for t in model_data.keys()}))
# a convenience for later use...
m2.tqa_idx = Set(within=m2.t*m2.q*m2.a, initialize=model_data.keys())
### PARAMS
m2.EERF = Param(m2.t, m2.q, m2.a, initialize=
{t[0]:t[1]['tons'] for t in model_data.items()})
print(' *** Concrete Model *** ')
m2.pprint()
1 Set Declarations
tqa : Size=1, Index=None, Ordered=Insertion
Key : Dimen : Domain : Size : Members
None : 3 : Any : 4 : {(2020, 'ca', 'herb'), (2020, 'ca', 'waste'), (2020, 'nv', 'waste'), (2019, 'az', 'herb')}
1 Param Declarations
EERF : Size=4, Index=tqa, Domain=Any, Default=None, Mutable=False
Key : Value
(2019, 'az', 'herb') : 5
(2020, 'ca', 'herb') : 2
(2020, 'ca', 'waste') : 3
(2020, 'nv', 'waste') : 4
2 Declarations: tqa EERF
*** Concrete Model ***
7 Set Declarations
EERF_index : Size=1, Index=None, Ordered=True
Key : Dimen : Domain : Size : Members
None : 3 : t*q*a : 12 : {(2019, 'az', 'herb'), (2019, 'az', 'waste'), (2019, 'ca', 'herb'), (2019, 'ca', 'waste'), (2019, 'nv', 'herb'), (2019, 'nv', 'waste'), (2020, 'az', 'herb'), (2020, 'az', 'waste'), (2020, 'ca', 'herb'), (2020, 'ca', 'waste'), (2020, 'nv', 'herb'), (2020, 'nv', 'waste')}
a : Size=1, Index=None, Ordered=Insertion
Key : Dimen : Domain : Size : Members
None : 1 : Any : 2 : {'herb', 'waste'}
q : Size=1, Index=None, Ordered=Insertion
Key : Dimen : Domain : Size : Members
None : 1 : Any : 3 : {'az', 'ca', 'nv'}
t : Size=1, Index=None, Ordered=Insertion
Key : Dimen : Domain : Size : Members
None : 1 : Any : 2 : {2019, 2020}
tqa_idx : Size=1, Index=None, Ordered=Insertion
Key : Dimen : Domain : Size : Members
None : 3 : tqa_idx_domain : 4 : {(2020, 'ca', 'herb'), (2020, 'ca', 'waste'), (2020, 'nv', 'waste'), (2019, 'az', 'herb')}
tqa_idx_domain : Size=1, Index=None, Ordered=True
Key : Dimen : Domain : Size : Members
None : 3 : tqa_idx_domain_index_0*a : 12 : {(2019, 'az', 'herb'), (2019, 'az', 'waste'), (2019, 'ca', 'herb'), (2019, 'ca', 'waste'), (2019, 'nv', 'herb'), (2019, 'nv', 'waste'), (2020, 'az', 'herb'), (2020, 'az', 'waste'), (2020, 'ca', 'herb'), (2020, 'ca', 'waste'), (2020, 'nv', 'herb'), (2020, 'nv', 'waste')}
tqa_idx_domain_index_0 : Size=1, Index=None, Ordered=True
Key : Dimen : Domain : Size : Members
None : 2 : t*q : 6 : {(2019, 'az'), (2019, 'ca'), (2019, 'nv'), (2020, 'az'), (2020, 'ca'), (2020, 'nv')}
1 Param Declarations
EERF : Size=4, Index=EERF_index, Domain=Any, Default=None, Mutable=False
Key : Value
(2019, 'az', 'herb') : 5
(2020, 'ca', 'herb') : 2
(2020, 'ca', 'waste') : 3
(2020, 'nv', 'waste') : 4