Developer Journal - March 2018
A reviewed monthly digest generated from public GitHub activity.
This developer-journal entry was generated with a local AI model from my public GitHub activity, then reviewed before publication.
Digest
March 2018 development focused on multi-threaded processing optimizations, library integration, and Spark job refinements. Key activities included implementing jblas for parallel calculations, replacing legacy Wigner symbol code with a Maven dependency, and adjusting Spark code to handle HDFS storage and YARN job submission.
main java codebase
Work in March 2018 centered on enhancing the Java codebase for climate data processing. Initial experiments with jblas for multi-threaded calculations were implemented, including a new multi-threaded processor class (SphericalHarmonics_LongTermStations_FullSpectral_multi.java). The project integrated the svengato/Wigner3j6j9j library via Maven, removing legacy Wigner symbol implementation files. Spark-related code was refactored to save intermediate results to HDFS for YARN compatibility and to address CLI version issues. Bug fixes were applied to PCA calculation logic and Spark job submission.
Notable sources
- https://github.com/dafrenchyman/mrsharky/commit/65e64408bd6de5b6305eb5ba9bba8d29f0e302df
- https://github.com/dafrenchyman/mrsharky/commit/09fa48ea61c7804bf900b8aff97c9e26f3dfdf91
- https://github.com/dafrenchyman/mrsharky/commit/7ce293ce8567e9d98ef24a4743b906249c11613b