Introduction

Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

Summary of the week

During this week, significant progress was made in the project, focusing on resolving dependencies and successfully compiling the NLU Intent Engine. The compilation of Scikit-Learn along with SciPy in a single layer was accomplished after making necessary modifications to address conflicts. However, runtime errors related to undefined symbols in SciPy were encountered. After extensive investigation, it was determined that these errors were caused by failed linking of OpenBlas libraries with SciPy. A hypothesis was formulated and tested by manually specifying the OpenBlas library using the LD_PRELOAD flag, which proved successful in running the intent engine on the target machine.

Tasks completed

• Made slight modifications to resolve conflicts and successfully compiled Scikit-Learn and SciPy in a single layer. The Lapack and OpenBlas libraries were having conflicts over a single header file. Modified the bbappend of Lapack by adding following function:

  do_install:append() {
      rm -f ${D}${includedir}/cblas.h
  }
  

• Ensured that all the necessary dependencies for the NLU Intent Engine were properly installed and configured.
• Encountered runtime errors related to undefined symbols in SciPy during execution.
• Debugged the issue to identify the root cause, which was found to be the failed linking of OpenBlas libraries with SciPy.
• Temporarily resolved the SciPy linking issue by manually specifying the OpenBlas library (located at: /usr/lib/libopenblas.so.0) using the LD_PRELOAD flag.

Tasks leftover

The following tasks were leftover this week:

• SciPy and OpenBlas linking errors need to be resolved.

Running NLU Intent Engine (Optional)

• Clone the updated meta-offline-voice-agent layer into meta-agl-devel.
• Update the templates/feature/agl-offline-voice-agent/50_local.conf.inc file with the following lines:

  # enabling fortran
  FORTRAN:forcevariable = ",fortran"
  RUNTIMETARGET:append:pn-gcc-runtime = " libquadmath"
  # bad practice but required to build scikit-learn
  HOSTTOOLS += "gfortran"
  # all the dependencies which get included in the final image
  IMAGE_INSTALL:append = " \
      python3-deprecation \
      python3-packaging \
      python3-num2words \
      python3-pyaml \
      python3-requests \
      python3-future \
      python3-scipy \
      python3-scikit-learn \
      python3-threadpoolctl \
      python3-python-crfsuite \
      python3-tabulate \
      python3-six \
      python3-tqdm \
      python3-sklearn-crfsuite \
      python3-snips-nlu-utils \
      python3-snips-nlu-parsers \
      python3-nlu-inference-agl \
      python3-vosk-api \
      vosk-kaldi \
      vosk vosk-server \
      python3-vosk-websocket-server \
      python3-sounddevice \
      flutter-vosk-demo \
      "
  

• Build the final image, preferably agl-demo-platform.
• After booting into the target image, execute the following command:

  $ export LD_PRELOAD=/usr/lib/libopenblas.so.0
  

• Clone the pre-trained model hosted on GitHub using the following commands:

  $ wget https://github.com/malik727/nlu-model-agl/archive/refs/heads/main.zip
  $ unzip main.zip
  $ cd nlu-model-agl-main
  

• Finally, load and test the NLU Intent Engine using the following command pattern:

  $ nlu-inference-agl parse /path/to/model-directory -q "your command here"
  

  For example:

  $ nlu-inference-agl parse model -q "roll down driver side window"
  

To learn more about the NLU Intent Engine model and the currently supported intents, visit the repository.

Next steps

• Resolve SciPy and OpenBlas Linking Issues: Continue investigating and addressing the root cause of the failed linking between SciPy and OpenBlas libraries.
• Upload to Gerrit: Add all the lastest work to Gerrit as a patch set after cleanup of unnecessary files.

Conclusion

Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

Resources

There were no resources found attached to this post.