palantir-foundry foundry-code-repositories

How do I improve the performance of my Code Assist in Code Repository?

Code Assist is very slow and takes up to an hour to resolve the environment. How would I improve the performance here?

Solution

Short answer: If there has been no commits made the past 7 days, the environment cache (used by Code Assist) may have expired. Try making a commit to make the CI recompute the environment cache.

Long answer: When working with python transforms, a valid python environment needs to be computed that satisfies all required packages and their dependencies. The more python packages that are required by the environment, and the more specific their versions, the harder (and thus longer) this problem becomes to solve. For this particular code repository, here are all the python packages and versions set as required in the meta.yaml file:

# If you need to modify the runtime requirements for your package,
# update the 'requirements.run' section in this file

package:
  name: "{{ PACKAGE_NAME }}"
  version: "{{ PACKAGE_VERSION }}"

source:
  path: ../src

requirements:
  build:
    - python 3.6.*
    - setuptools
    - unionutils 0.0.39
    - palantirutils 0.1.100

  # Any extra packages required to run your package.
  run:
    - python 3.6.*
    - transforms {{ PYTHON_TRANSFORMS_VERSION }}
    - unionutils 0.0.39
    - palantirutils 0.1.100
    - transforms-expectations

build:
  script: python setup.py install --single-version-externally-managed --record=record.txt

In practice though, these packages have their own dependencies, and so the resolved environment for this repo actually requires a lot more different python packages with specific versions (you can see these in the conda lock file which is a hidden file in the repo). In this case there are 123 additional dependencies.

1. # This file is autogenerated and supposed to be committed to the git repository.
2. # Changing this file is not recommended and will likely break the build.
3. # Run `./gradlew --write-locks` to regenerate.
4. _libgcc_mutex=0.1=conda_forge @ linux-64
5. _openmp_mutex=4.5=2_gnu @ linux-64
6. alsa-lib=1.2.7.2=h166bdaf_0 @ linux-64
7. brotlipy=0.7.0=py36h27cfd23_1003 @ linux-64
8. ca-certificates=2022.12.7=ha878542_0 @ linux-64
9. certifi=2021.5.30=py36h5fab9bb_0 @ linux-64
10. cffi=1.14.6=py36hd8eec40_1 @ linux-64
11. charset-normalizer=2.1.1=pyhd8ed1ab_0 @ noarch
12. cryptography=35.0.0=py36hd23ed53_0 @ linux-64
13. cycler=0.11.0=pyhd8ed1ab_0 @ noarch
14. dbus=1.13.18=hb2f20db_0 @ linux-64
15. decorator=5.1.1=pyhd3eb1b0_0 @ noarch
16. expat=2.5.0=h27087fc_0 @ linux-64
17. font-ttf-dejavu-sans-mono=2.37=h6964260_0 @ noarch
18. font-ttf-inconsolata=3.000=h77eed37_0 @ noarch
19. font-ttf-source-code-pro=2.038=h77eed37_0 @ noarch
20. font-ttf-ubuntu=0.83=h8b1ccd4_0 @ noarch
21. fontconfig=2.14.1=hc2a2eb6_0 @ linux-64
22. fonts-anaconda=1=h8fa9717_0 @ noarch
23. fonts-conda-ecosystem=1=hd3eb1b0_0 @ noarch
24. freetype=2.12.1=hca18f0e_1 @ linux-64
25. freezegun=1.2.2=pyhd8ed1ab_0 @ noarch
26. future=0.18.2=py36h5fab9bb_3 @ linux-64
27. gettext=0.21.1=h27087fc_0 @ linux-64
28. glib-tools=2.72.1=h6239696_0 @ linux-64
29. glib=2.72.1=h6239696_0 @ linux-64
30. gst-plugins-base=1.20.3=h57caac4_2 @ linux-64
31. gstreamer=1.20.3=hd4edc92_2 @ linux-64
32. icu=69.1=h9c3ff4c_0 @ linux-64
33. idna=3.4=pyhd8ed1ab_0 @ noarch
34. jpeg=9e=h166bdaf_2 @ linux-64
35. keyutils=1.6.1=h166bdaf_0 @ linux-64
36. kiwisolver=1.3.1=py36h605e78d_1 @ linux-64
37. krb5=1.20.1=hf9c8cef_0 @ linux-64
38. lcms2=2.14=hfd0df8a_1 @ linux-64
39. ld_impl_linux-64=2.39=hcc3a1bd_1 @ linux-64
40. lerc=4.0.0=h27087fc_0 @ linux-64
41. libblas=3.9.0=16_linux64_openblas @ linux-64
42. libcblas=3.9.0=16_linux64_openblas @ linux-64
43. libclang=13.0.1=default_hc23dcda_0 @ linux-64
44. libdeflate=1.14=h166bdaf_0 @ linux-64
45. libedit=3.1.20210714=h7f8727e_0 @ linux-64
46. libevent=2.1.10=h9b69904_4 @ linux-64
47. libffi=3.4.2=h6a678d5_6 @ linux-64
48. libgcc-ng=12.1.0=h8d9b700_17 @ linux-64
49. libgfortran-ng=12.2.0=h69a702a_19 @ linux-64
50. libgfortran5=12.2.0=h337968e_19 @ linux-64
51. libglib=2.72.1=h2d90d5f_0 @ linux-64
52. libgomp=12.1.0=h8d9b700_17 @ linux-64
53. libiconv=1.16=h516909a_0 @ linux-64
54. liblapack=3.9.0=16_linux64_openblas @ linux-64
55. libllvm13=13.0.1=hf817b99_2 @ linux-64
56. libnsl=2.0.0=h7f98852_0 @ linux-64
57. libogg=1.3.5=h27cfd23_1 @ linux-64
58. libopenblas=0.3.21=pthreads_h78a6416_3 @ linux-64
59. libopus=1.3.1=h7f98852_1 @ linux-64
60. libpng=1.6.39=h753d276_0 @ linux-64
61. libpq=14.5=h2baec63_4 @ linux-64
62. libsqlite=3.40.0=h753d276_0 @ linux-64
63. libstdcxx-ng=12.1.0=ha89aaad_17 @ linux-64
64. libtiff=4.5.0=h82bc61c_0 @ linux-64
65. libuuid=2.32.1=h14c3975_1000 @ linux-64
66. libvorbis=1.3.7=he1b5a44_0 @ linux-64
67. libwebp-base=1.2.4=h166bdaf_0 @ linux-64
68. libxcb=1.13=h7f98852_1004 @ linux-64
69. libxkbcommon=1.0.3=he3ba5ed_0 @ linux-64
70. libxml2=2.9.12=h885dcf4_1 @ linux-64
71. libzlib=1.2.13=h166bdaf_4 @ linux-64
72. lz4-c=1.9.3=h295c915_1 @ linux-64
73. matplotlib-base=3.3.4=py36hd391965_0 @ linux-64
74. matplotlib=3.3.4=py36h5fab9bb_0 @ linux-64
75. mysql-common=8.0.31=haf5c9bc_0 @ linux-64
76. mysql-libs=8.0.31=h28c427c_0 @ linux-64
77. ncurses=6.2=h58526e2_4 @ linux-64
78. nspr=4.35=h27087fc_0 @ linux-64
79. nss=3.82=he02c5a1_0 @ linux-64
80. numpy=1.19.5=py36hfc0c790_2 @ linux-64
81. olefile=0.46=pyh9f0ad1d_1 @ noarch
82. openjpeg=2.4.0=hb52868f_1 @ linux-64
83. openssl=1.1.1=h7b6447c_0 @ linux-64
84. palantir-spark-time=3.15.0=py_0 @ noarch
85. palantirutils=0.1.100=py36_0 @ linux-64
86. pandas=1.1.5=py36ha9443f7_0 @ linux-64
87. pcre=8.45=h9c3ff4c_0 @ linux-64
88. pillow=8.3.2=py36h676a545_0 @ linux-64
89. pip=22.0.2=pyhd8ed1ab_0 @ noarch
90. pthread-stubs=0.4=h36c2ea0_1001 @ linux-64
91. py4j=0.10.9.7=pyhd8ed1ab_0 @ noarch
92. pycparser=2.21=pyhd3eb1b0_0 @ noarch
93. pydicom=2.3.1=pyh1a96a4e_0 @ noarch
94. pyopenssl=22.0.0=pyhd8ed1ab_1 @ noarch
95. pyparsing=3.0.9=pyhd8ed1ab_0 @ noarch
96. pyqt-impl=5.12.3=py36h7ec31b9_7 @ linux-64
97. pyqt5-sip=4.19.18=py36hc4f0c31_7 @ linux-64
98. pyqt=5.12.3=py36h5fab9bb_7 @ linux-64
99. pyqtchart=5.12=py36h7ec31b9_7 @ linux-64
100. pyqtwebengine=5.12.1=py36h7ec31b9_7 @ linux-64
101. pysocks=1.7.1=py36h5fab9bb_3 @ linux-64
102. pyspark-src=3.2.1_palantir.30=py_0 @ noarch
103. pyspark=3.2.1_palantir.30=py_0 @ noarch
104. python-dateutil=2.8.2=pyhd3eb1b0_0 @ noarch
105. python=3.6.15=hb7a2778_0_cpython @ linux-64
106. python_abi=3.6=2_cp36m @ linux-64
107. pytz=2022.7=pyhd8ed1ab_0 @ noarch
108. qt=5.12.9=h1304e3e_6 @ linux-64
109. readline=8.1=h27cfd23_0 @ linux-64
110. requests=2.28.1=pyhd8ed1ab_0 @ noarch
111. sas7bdat=2.2.3=pyhd8ed1ab_0 @ noarch
112. setuptools=58.0.4=py36h5fab9bb_2 @ linux-64
113. six=1.16.0=pyhd3eb1b0_1 @ noarch
114. sqlite=3.40.0=h5082296_0 @ linux-64
115. tk=8.6.12=h27826a3_0 @ linux-64
116. tornado=6.1=py36h8f6f2f9_1 @ linux-64
117. transforms-expectations=0.153.0=py_0 @ noarch
118. transforms=1.575.0=py_0 @ noarch
119. unionutils=0.0.39=py36_0 @ linux-64
120. urllib3=1.26.13=pyhd8ed1ab_0 @ noarch
121. wheel=0.37.1=pyhd3eb1b0_0 @ noarch
122. xorg-libxau=1.0.9=h7f98852_0 @ linux-64
123. xorg-libxdmcp=1.1.3=h516909a_0 @ linux-64
124. xz=5.2.9=h166bdaf_0 @ linux-64
125. zlib=1.2.13=h166bdaf_4 @ linux-64
126. zstd=1.5.2=h8a70e8d_4 @ linux-64

[Conda lock version]
v2 - fc1db1a3046b71820e26b48a2910180bd17d802db0610c30351282a31083fe9b

When the first commit is made to a new python repo there are two relevant tasks that take place during the continuous integration (CI) checks. The first task solves the python environment by calculating a mix of packages that satisfies all the dependencies and requirements of the repo, and the second one then downloads these specific package versions. This resolved environment is recorded in the hidden conda lock file, and is valid for 7 days (essentially it is cached).

Any time a new commit is made in the following 7 days after an environment solve, the CI checks can use the cached python environment to skip those two tasks of resolving the environment and downloading the packages, which dramatically speeds up the CI check time.

If however the commit is made after the cache is 7 days old, or if a commit is made that changes the required python packages or versions in the meta.yaml file, then the environment will need to be solved again, which will lead to a longer CI time.

So how does this relate to code assist?

Well, code assist essentially needs to run the same tasks, but can also read from that cache file. So if you notice that code assist is taking a long time to run, that's because it is not able to read from the cache (either because of changed packages or cache expiry) and is instead having to solve the environment itself. However, as code assist is not allowed to write code to your repository it cannot modify the cache after it has finished its solve.

So in this case I would recommend making a commit to the code of the repo to reset the cache, and then you should notice that code assist starts a lot faster than 40 minutes (until the 7 day cache expires or the packages change).