Search code examples
Haskell's double behaviour...

haskelldoubleprecision

Read More
c++ long double (128-bit) precision...

x86-64precisionc++23long-double128-bit

Read More
What class to use for money representation?...

pythoncurrencyrounding-errorarbitrary-precisionprecision

Read More
efficient cumulative log_sum_exp...

rprecision

Read More
Why 5.0 / 2 returns 2.5000000000000000 (scale of 16) instead of 2.5 (scale of 1)?...

sqlpostgresqlprecisionnumericarbitrary-precision

Read More
float is overflowing in openGL despite being under 32 bits...

c++openglfloating-pointglslprecision

Read More
Intel x86_64 assembly compare signed double precision floats...

assemblyx86-64intelprecisionsse

Read More
How many digits can I rely on?...

floating-pointprecisionnumerical-computing

Read More
How to choose epsilon value for floating point?...

mathfloating-pointprecisionepsilon

Read More
synopsys dc_shell get_attribute number of digits...

tclprecision

Read More
How can i make my math more stable for large circle radii...

c#mathgeometryprecision

Read More
Java Double Precision - Rounding - %f specifier...

javafloating-pointprecisionieee-754

Read More
How to properly round-up half float numbers?...

pythonpython-3.xfloating-pointroundingprecision

Read More
BigDecimal, precision and scale...

javajpascaleprecisionbigdecimal

Read More
Has anyone encounter this this stripping artifact during "RayTracing in one Weekend"?...

randomprecisionraytracingcompute-shader

Read More
Strange difference of result when in a for loop in R...

rloopsfor-loopprecisiongeostatistics

Read More
Specify number length and precision in materialized view, starting from a pivot...

oracle-databasepivotprecisionnumericmaterialized-views

Read More
Floating Point: Distance between MAX and INF...

floating-pointprecision

Read More
What is the precision of long double in C++?...

c++precisionlong-double

Read More
get float64 for numpy calculation...

numpydecimalprecision

Read More
Is floating-point math broken?...

mathfloating-pointlanguage-agnosticprecisionfloating-accuracy

Read More
Is there any way to get current time in nanoseconds using JavaScript?...

javascripttimeprecisiontime-precision

Read More
What role does min value and max value play in reducing memory usage?...

pythonpandasnumpymemoryprecision

Read More
80 bit floating point arithmetic in C/C++...

cprecision

Read More
C precision of double: compiler dependent?...

cprecision

Read More
Mystery behind presentation of Floating Point numbers...

swiftfloating-pointdecimalprecisionfloating-accuracy

Read More
Example of Code with and without strictfp Modifier...

javafloating-pointprecisionieee-754strictfp

Read More
Format floating-point numbers with sed to equal precision - add trailing zeros...

bashsedfloating-pointprecisionfloating-accuracy

Read More
Why is this result more accurate than another result by equivalent functions...

pythonfloating-pointprecision

Read More
Possible loss of precision during conversion of float number...

javadecimalprecision

Read More
BackNext