I am trying to use statsmodel's nonparametric lowess regression.
import statsmodels.api as sm
lowess = sm.nonparametric.lowess
x = # see array below
y = # array below
The two arrays I'm regressing on look like:
y = array([0.39460831, 0.3943878 , 0.42335204, 0.45601645, 0.35705666,
0.43106245, 0.40876906, 0.40865107, 0.44079581, 0.40809921,
0.34715472, 0.28673869, 0.41429625, 0.33814808, 0.36065362,
0.42323234, 0.43075707, 0.39036774, 0.45624754, 0.4264153 ,
0.40854912, 0.33814808, 0.38919367, 0.47063437, 0.41819225,
0.43655294, 0.39817544, 0.41817408, 0.38579407, 0.3758216 ,
0.39295075, 0.35279793, 0.41396078, 0.35835224, 0.39877632,
0.41840176, 0.42235122, 0.31909751, 0.41690713, 0.40415778,
0.36172602, 0.34810291, 0.45065063, 0.44980572, 0.4362235 ,
0.4041436 , 0.34270833, 0.45133976, 0.43398212, 0.4352659 ,
0.27949522, 0.47130217, 0.35789634, 0.35249196, 0.35665419,
0.45161807, 0.43424503, 0.35939123, 0.45345458, 0.3410767 ,
0.43481544, 0.33007379, 0.31909751, 0.27520451, 0.35824199,
0.35671327, 0.39329133, 0.3978407 , 0.4264153 , 0.38899924,
0.44263212, 0.3758216 , 0.36393358, 0.38579407, 0.26942863,
0.47378067, 0.41692978, 0.31262075, 0.28673869, 0.45270572,
0.45358348, 0.33007379, 0.46485582, 0.38381883, 0.40847049,
0.4552592 , 0.27520451, 0.33233801, 0.37616339, 0.45312021,
0.39295075, 0.34311701, 0.45405103, 0.33026848, 0.35512348,
0.35781538, 0.34892051, 0.39946672, 0.33814808, 0.45352116,
0.3457032 , 0.39038526, 0.31909751, 0.4362235 , 0.45294301,
0.33150022, 0.41658717, 0.45179918, 0.44187217, 0.45275743,
0.42749683, 0.41658717, 0.43431399, 0.43431399, 0.41056891,
0.4362235 , 0.34365128, 0.3758216 , 0.31107728, 0.45103128,
0.39817544, 0.33026848, 0.45603937, 0.45043726, 0.27949522,
0.44079581, 0.36393358, 0.44908607, 0.44996669, 0.44877551,
0.34715472, 0.2674405 , 0.4367155 , 0.37446672, 0.29808357,
0.41506218, 0.34848782, 0.39817544, 0.26261556, 0.37446672,
0.3925812 , 0.35279793, 0.38381883, 0.39168392, 0.39917386,
0.43770031, 0.44263212, 0.3410767 , 0.34058531, 0.3457032 ,
0.43773453, 0.33814808, 0.39221085, 0.45325688, 0.44079581,
0.38579407, 0.39165179, 0.38675654, 0.33180298, 0.42533256,
0.38089742, 0.33567492, 0.45202054, 0.46304505, 0.35671327,
0.24622921, 0.35665419, 0.38733682, 0.4264153 , 0.43539081,
0.29633964, 0.37827505, 0.38701604, 0.35939123, 0.38291542,
0.4413525 , 0.43415077, 0.19365321, 0.26597312, 0.35705666,
0.46805815, 0.38402756, 0.344355 , 0.39295075, 0.38900422,
0.34102323, 0.45161807, 0.43035007, 0.30790008, 0.43431399,
0.30624349, 0.45190784, 0.33567492, 0.344355 , 0.43075707,
0.41690713, 0.44908607, 0.32628674, 0.41108733, 0.41423588,
0.3024584 , 0.34102323, 0.41692978, 0.42424687, 0.34270833,
0.34058531, 0.26391213, 0.41108733, 0.43106245, 0.33233801,
0.33814808, 0.31816119, 0.36172602, 0.36106708, 0.37616339,
0.1717933 , 0.29266398, 0.28673869, 0.25844219, 0.46372781,
0.35835224, 0.46637148, 0.3663237 , 0.45297022, 0.44975711,
0.28564902, 0.42772618, 0.41056891, 0.33814808, 0.37050513,
0.26967221, 0.32682098, 0.40847049, 0.31635641, 0.31909751,
0.33347002, 0.32393267, 0.37002245, 0.32407268, 0.41396078,
0.34365128, 0.41429625, 0.41696097, 0.32144044, 0.44908607,
0.32856049, 0.33036114, 0.27520451, 0.31909751, 0.27949522,
0.30185272, 0.35824199, 0.40384943, 0.36065362, 0.35919559,
0.23375244, 0.45161807, 0.43106245, 0.31330424, 0.31227119,
0.42390002, 0.44658582, 0.18798846, 0.47670001, 0.47980779,
0.40565634, 0.2525325 , 0.41990849, 0.44037665, 0.32682098,
0.45987354, 0.25672615, 0.23839431, 0.27520451, 0.44263212,
0.43833015, 0.42335204, 0.3100529 , 0.42323234, 0.31093935,
0.28673869, 0.23839431, 0.44371944, 0.2827502 , 0.39270912,
0.42325646, 0.35252627, 0.36903986, 0.33007379, 0.38381883,
0.46089718, 0.28396175, 0.43911932, 0.26672449, 0.3410767 ,
0.41892012, 0.1717933 , 0.30693675, 0.33814808, 0.43318908,
0.35824199, 0.37950292, 0.46129589, 0.25452025, 0.23375244,
0.30712549, 0.41901032, 0.36828848, 0.30788856, 0.32856049,
0.36771637, 0.26494857, 0.47713128, 0.47389691, 0.27949522,
0.31507796, 0.33036114, 0.32628674, 0.30185272, 0.31909751,
0.29791308, 0.3593851 , 0.24622921, 0.37455814, 0.26967221,
0.28673869, 0.30016467, 0.27520451, 0.38229718, 0.25025446,
0.33180298, 0.31816119, 0.45963692, 0.31635641, 0.45522095,
0.32985042, 0.32144044, 0.3024584 , 0.34715472, 0.43283525,
0.35705666, 0.31217255, 0.32407268, 0.41351679, 0.34508365,
0.30047853, 0.30118257, 0.37682165, 0.36393358, 0.32832871,
0.27933756, 0.37446672, 0.43506952, 0.31816119, 0.31635641,
0.35988311, 0.35671327, 0.32919213, 0.39700601, 0.17783712,
0.34058531, 0.33036114, 0.32856049, 0.25290452, 0.30147635,
0.43682787, 0.32682098, 0.3481169 , 0.36671704, 0.32144044,
0.28059754, 0.46192216, 0.34731488, 0.26537987, 0.34751195,
0.26354371, 0.38429764, 0.34672167, 0.40366341, 0.33180298,
0.36729379, 0.43119706, 0.34186555, 0.29266398, 0.39534354,
0.29777454, 0.36771012, 0.25844219, 0.27949522, 0.39062032,
0.32407268, 0.35939123, 0.28673869, 0.45505276, 0.34036781,
0.30624349, 0.30790008, 0.27520451, 0.34268798, 0.37616339,
0.26261556, 0.24622921, 0.30934533, 0.30802175, 0.30185272,
0.34121105, 0.29611571, 0.35835224, 0.2990357 , 0.33412808,
0.40805029, 0.31330424, 0.33150022, 0.33208352, 0.33150022,
0.3091976 , 0.31022855, 0.29812804, 0.30051468, 0.30051468,
0.29812804, 0.27949522, 0.37954714, 0.34171002, 0.31469502,
0.38749887, 0.33347002, 0.32393267, 0.34095206, 0.31293642,
0.32393267, 0.30649068, 0.38043712, 0.33196421, 0.33587843,
0.25991555, 0.31701787, 0.31000817, 0.3177085 , 0.25844219,
0.37388784, 0.26967221, 0.30877847, 0.3457032 , 0.33347002,
0.35030022, 0.344355 , 0.31507796, 0.27287812, 0.3431596 ,
0.35110244, 0.34890499, 0.3495343 , 0.35279793, 0.34234006,
0.44562415, 0.30204603, 0.23839431, 0.31174037, 0.25991555,
0.29278232, 0.31227119, 0.29226819, 0.1717933 , 0.34365128,
0.32403962, 0.31514662, 0.32231812, 0.3897924 , 0.29584331,
0.23375244, 0.36638174, 0.30530051, 0.36568886, 0.35720481,
0.30552954, 0.35051081, 0.35642678, 0.29582181, 0.30185272,
0.26354371, 0.26261556, 0.36172602, 0.32978801, 0.35005709,
0.27489794, 0.36065362, 0.33016554, 0.23375244, 0.31645336,
0.29808357, 0.34399874, 0.29633964, 0.24622921, 0.31514662,
0.28564902, 0.31262075, 0.30552954, 0.31107728, 0.27676855,
0.31507796, 0.31514662, 0.35875935, 0.25025446, 0.3608851 ,
0.35956166, 0.26537987, 0.30051468, 0.28778531, 0.30185272,
0.26424722, 0.26494857, 0.31621874, 0.45565476, 0.45541966,
0.34187107, 0.45180643, 0.46396924, 0.28593057, 0.46273672,
0.4631243 , 0.27556483, 0.46390976, 0.45542472, 0.45565796,
0.34237884, 0.31227119, 0.46306743, 0.30185272, 0.1717933 ,
0.37166394, 0.23839431, 0.18798846, 0.27489794, 0.27556483,
0.25452025, 0.27520451, 0.37553933, 0.26261556, 0.46451886,
0.26424722, 0.26967221, 0.46732794, 0.26672449, 0.29812804,
0.46331317, 0.46787095, 0.26391213, 0.4561402 , 0.30185272,
0.46620574, 0.2674405 , 0.41658717, 0.41690713, 0.40847049,
0.27949522, 0.2525325 , 0.46676643, 0.31909751, 0.35720607,
0.41056891, 0.38720735, 0.46666564, 0.25844219, 0.41108733,
0.35809937, 0.41396078, 0.33233801, 0.38879081, 0.27676855,
0.33567492, 0.43231008, 0.43338316, 0.32628674, 0.2767611 ,
0.41429625, 0.28564902, 0.42323234, 0.28673869, 0.31506268,
0.43249034, 0.42334298, 0.43075707, 0.31330424, 0.41633699,
0.27455114, 0.25290452, 0.42335204, 0.25672615, 0.31500147,
0.42727391, 0.41692978, 0.43358162, 0.43269304, 0.37385574,
0.27834975, 0.43168928, 0.26942863, 0.19365321, 0.42550945,
0.43847614, 0.26597312, 0.42730983, 0.37523301, 0.30552954,
0.42752101, 0.42323596, 0.33026848, 0.34102323, 0.35279942,
0.43681325, 0.29633964, 0.3589688 , 0.23375244, 0.42756389,
0.17783712, 0.33666363, 0.34794648, 0.24622921, 0.35665419,
0.3024584 , 0.34270833, 0.35665194, 0.26856497, 0.34522123,
0.30624349, 0.26620059, 0.26967221, 0.29266398, 0.43770414,
0.25844219, 0.31107728, 0.36892801, 0.29808357, 0.25844219,
0.3282393 , 0.3285013 , 0.43613956, 0.26967221, 0.1717933 ,
0.26261556, 0.30790008, 0.43342386, 0.31262075, 0.26261556,
0.4316186 , 0.19697687, 0.23375244, 0.43168928, 0.1717933 ,
0.23839431, 0.18798846, 0.18798846, 0.23839431, 0.1717933 ,
0.25991555, 0.23229651, 0.1717933 , 0.19365321, 0.24622921,
0.26967221, 0.1717933 , 0.25844219, 0.36771552, 0.18798846,
0.1717933 , 0.3666904 , 0.1717933 , 0.1717933 , 0.1717933 ,
0.26537987, 0.36076434, 0.26354371, 0.17783712, 0.35960867,
0.19697687, 0.27556483, 0.26261556, 0.19365321, 0.19365321,
0.26672449, 0.26494857, 0.17783712, 0.27676855, 0.27489794,
0.26424722, 0.17783712, 0.1717933 , 0.18798846, 0.32850539,
0.1717933 , 0.23375244, 0.17783712, 0.2525325 , 0.23839431,
0.19365321, 0.17783712, 0.24622921, 0.26391213, 0.18798846,
0.19365321, 0.2674405 , 0.25025446, 0.27185207, 0.25452025,
0.26597312, 0.25290452, 0.26942863, 0.25672615, 0.19697687])
x = array([5, 4, 5, 5, 5, 5, 4, 5, 5, 3, 4, 5, 4, 5, 5, 4, 5, 4, 4, 5, 5, 5,
5, 4, 4, 4, 5, 5, 4, 5, 5, 5, 5, 4, 5, 3, 5, 5, 5, 5, 4, 3, 5, 4,
4, 5, 4, 3, 5, 5, 5, 5, 5, 4, 5, 5, 4, 5, 5, 4, 5, 5, 5, 4, 5, 3,
4, 5, 5, 5, 5, 5, 5, 4, 5, 3, 5, 5, 5, 4, 5, 5, 5, 5, 3, 4, 4, 5,
5, 5, 5, 5, 4, 5, 5, 5, 4, 3, 5, 5, 5, 5, 5, 4, 5, 5, 4, 2, 5, 5,
5, 4, 5, 5, 5, 4, 4, 5, 4, 4, 5, 5, 5, 4, 5, 5, 5, 4, 5, 5, 4, 4,
5, 4, 4, 5, 4, 5, 5, 4, 5, 5, 5, 4, 5, 4, 5, 4, 5, 5, 5, 5, 5, 3,
5, 4, 5, 4, 4, 4, 3, 4, 5, 3, 3, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
3, 5, 4, 5, 5, 4, 5, 5, 5, 5, 5, 4, 4, 5, 5, 5, 4, 5, 5, 5, 4, 5,
4, 5, 3, 5, 5, 5, 4, 5, 5, 4, 5, 5, 5, 5, 4, 5, 5, 5, 5, 5, 4, 5,
4, 4, 4, 4, 5, 3, 5, 5, 5, 5, 5, 5, 3, 4, 5, 5, 4, 4, 5, 5, 4, 4,
5, 3, 4, 5, 4, 4, 5, 5, 5, 5, 5, 5, 5, 4, 5, 5, 5, 5, 5, 4, 4, 5,
4, 5, 3, 3, 4, 5, 3, 4, 5, 4, 5, 5, 5, 3, 4, 5, 5, 5, 5, 4, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 4, 5, 5, 4, 5, 4, 5, 5, 5, 5, 4, 5, 4, 4,
5, 5, 5, 4, 5, 4, 5, 5, 4, 5, 5, 5, 2, 4, 5, 4, 5, 5, 5, 4, 5, 3,
4, 5, 4, 4, 5, 5, 3, 3, 4, 4, 5, 4, 5, 4, 2, 4, 4, 5, 5, 4, 5, 4,
5, 5, 4, 5, 3, 5, 5, 4, 5, 4, 5, 5, 5, 4, 5, 4, 5, 3, 4, 5, 5, 4,
5, 5, 4, 4, 4, 4, 5, 4, 4, 5, 4, 5, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5,
4, 4, 5, 5, 5, 5, 5, 4, 5, 4, 3, 4, 4, 2, 5, 5, 5, 5, 5, 3, 3, 5,
4, 4, 5, 5, 4, 5, 5, 5, 5, 4, 3, 5, 4, 3, 5, 5, 5, 3, 5, 5, 5, 4,
5, 5, 4, 5, 5, 5, 5, 5, 3, 5, 5, 5, 4, 5, 4, 5, 4, 5, 5, 3, 5, 5,
5, 5, 4, 4, 4, 4, 5, 5, 4, 5, 5, 4, 4, 5, 4, 5, 4, 5, 5, 5, 4, 5,
2, 5, 5, 5, 4, 5, 4, 3, 5, 5, 4, 3, 5, 5, 4, 4, 5, 4, 5, 3, 5, 3,
4, 4, 5, 5, 3, 4, 3, 5, 4, 5, 2, 3, 4, 5, 4, 5, 5, 5, 4, 4, 5, 5,
5, 5, 5, 5, 4, 5, 5, 5, 4, 3, 5, 4, 3, 5, 4, 5, 5, 5, 5, 5, 3, 5,
5, 4, 4, 5, 3, 5, 3, 4, 5, 5, 5, 5, 3, 4, 4, 4, 5, 5, 4, 4, 4, 5,
5, 5, 5, 4, 3, 4, 5, 5, 4, 5, 5, 5, 5, 4, 5, 5, 4, 4, 4, 5, 3, 5,
4, 3, 5, 5, 5, 2, 4, 4, 5, 5, 5, 5, 4, 5, 5, 5, 5, 5, 5, 4, 4, 4,
5, 4, 5, 5, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 4, 4, 3, 4, 4, 4, 5, 5,
5, 5, 5, 4, 3, 5, 5, 5, 3, 4, 4, 5, 5, 4, 4, 5, 5, 3, 2, 5, 5, 5,
5, 5, 4, 5, 4, 5, 4, 5, 5, 5, 4, 4, 5, 4, 5, 3, 5, 5, 5, 5, 5, 4,
4, 4, 5, 3, 4, 5, 4, 2, 5, 4, 4, 3, 5, 5, 4, 5, 5, 4, 5, 4, 3, 2,
5, 4, 5, 5, 4, 3])
When I run the lowess on these though, I am getting the warning
lowess(t5.F1.values, t5.nterms.values)
/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.py:3405: RuntimeWarning: Invalid value encountered in median
r = func(a, **kwargs)
/anaconda3/lib/python3.7/site-packages/statsmodels/nonparametric/smoothers_lowess.py:165: RuntimeWarning: invalid value encountered in greater_equal
res = _lowess(y, x, frac=frac, it=it, delta=delta)
and the output looks like:
array([[ 2., nan],
[ 2., nan],
[ 2., nan],
...,
[ 5., nan],
[ 5., nan],
[ 5., nan]])
Not really sure why this isn't working? (I could do this in R, but I would prefer not to :-)
statsmodels lowess is not designed for the case with many identical x values. See https://github.com/statsmodels/statsmodels/issues/2449
The problem is in computing the set of neighbors which does not work in a way that would be appropriate for the case with many tied x values. Lowess was designed for continuous variables with no or only a few ties.
With just a few unique x-values a boxplot or similar would be more appropriate.