Personal Blog

Malik Talha


# AGL Introduction

> Automotive Grade Linux (AGL) is an open-source software platform specifically designed for automotive applications. It provides a standardized and customizable solution for connected cars and infotainment systems. AGL offers a comprehensive set of features, including a Linux-based operating system, middleware, and application framework. It enables automotive manufacturers and developers to collaborate, accelerate innovation, and create secure and reliable software for vehicles. AGL fosters an ecosystem of industry stakeholders, promoting interoperability and driving the adoption of open-source software in the automotive industry.

# Project Overview
My project revolves around enhancing the pre-existing speech recognition framework within the context of Automotive Grade Linux (AGL). The primary objective of this undertaking is the development and seamless integration of a sophisticated natural language intent engine. This engine is designed to comprehensively understand the underlying intent behind a user's voice command and facilitate the execution of said intent through a software agent. This ambitious endeavor represents a significant leap forward in the realm of in-vehicle voice interaction. Consequently, the project has been aptly named as the "Voice Assistant."

The Voice Assistant, a multifaceted project, combines cutting-edge speech recognition technologies and natural language processing capabilities to create a comprehensive voice-driven experience within automotive environments. It encompasses the ability to interpret a wide range of user commands, from basic vehicle control functions to more complex queries and requests, such as navigation, media control, and in-depth vehicle status inquiries. The aim of this project is to provide a seamless and intuitive voice interaction system that enhances the in-car experience, ultimately improving safety, convenience, and user satisfaction for AGL-equipped vehicles.

Interested in more technicals details and how to use the voice assistant? Check out the [**Official Automotive Grade Linux Documentation**](https://docs.automotivelinux.org/en/master/#06_Component_Documentation/10_agl_voice_agent_assistant/).

# Project Architecture
The following diagram shows the architecture of the project:

![Project Architecture](https://blog.maliktalha.me/images/agl_voice_agent_architecture.png)

# Planned Goals and Milestones
As per my GSoC project I planned to deliver the following goals and milestones:

- 🔖 **Objective # 01:** Integration and development of Natural Language Understanding Intent Engine with existing Vosk implementation.
- 🔖 **Objective # 02:** Ability to execute the intent extracted from the NLU engine using some sort of interface that will communicate with the APIs.

# Deliverables
I was able to successfully deliver all the planned goals and milestones. The following are the deliverables of my GSoC project:

- ✅ **Deliverable # 01:** Integration and development of **Snips** and **RASA** Natural Language Understanding Intent Engines.
- ✅ **Deliverable # 02:** Ability to execute the intent extracted from the NLU engine using a **Python** and **GStreamer** based voice agent service. 
- ✅ **Deliverable # 03:** A flutter based IVI application that communicates with the voice agent service to record user voice command, transcribe it to text, extract intent from it, and execute the intent.

# Project Gerrit Repositories
I have created and/or worked on the following AGL Gerrit repositories for my GSoC project:
- [**AGL/meta-agl-devel**](https://gerrit.automotivelinux.org/gerrit/q/owner:talhamalik727x@gmail.com+repo:AGL/meta-agl-devel)
- [**src/snips-inference-agl**](https://gerrit.automotivelinux.org/gerrit/q/owner:talhamalik727x@gmail.com+repo:src/snips-inference-agl)
- [**src/rasa-model-agl**](https://gerrit.automotivelinux.org/gerrit/q/owner:talhamalik727x@gmail.com+repo:src/rasa-model-agl)
- [**src/snips-model-agl**](https://gerrit.automotivelinux.org/gerrit/q/owner:talhamalik727x@gmail.com+repo:src/snips-model-agl)
- [**apps/agl-service-voiceagent**](https://gerrit.automotivelinux.org/gerrit/q/owner:talhamalik727x@gmail.com+repo:apps/agl-service-voiceagent)
- [**apps/flutter-speechrecognition-demo**](https://gerrit.automotivelinux.org/gerrit/q/owner:talhamalik727x@gmail.com+repo:apps/flutter-speechrecognition-demo)
- [**AGL/documentation**](https://gerrit.automotivelinux.org/gerrit/q/owner:talhamalik727x@gmail.com+repo:AGL/documentation)
- [**AGL/AGL-repo**](https://gerrit.automotivelinux.org/gerrit/q/owner:talhamalik727x@gmail.com+repo:AGL/AGL-repo)

# Gerrit Individual Commits
I have compiled the list of all the individual commits that I have made during my GSoC project on Gerrit. The following are the links to the commits:
- [**\#29191: Add gRPC-based voice agent service for AGL**](https://gerrit.automotivelinux.org/gerrit/c/apps/agl-service-voiceagent/+/29191) | apps/agl-service-voiceagent
- [**\#29251: Add dependencies for intent engine snips**](https://gerrit.automotivelinux.org/gerrit/c/AGL/meta-agl-devel/+/29251) | AGL/meta-agl-devel
- [**\#29265: Fix Scipy, OpenBlas, and NumPy library linking issues**](https://gerrit.automotivelinux.org/gerrit/c/AGL/meta-agl-devel/+/29265) | AGL/meta-agl-devel
- [**\#29266: Add dependencies for RASA framework**](https://gerrit.automotivelinux.org/gerrit/c/AGL/meta-agl-devel/+/29266) | AGL/meta-agl-devel
- [**\#29272: Add meta-tensorflow layer to external**](https://gerrit.automotivelinux.org/gerrit/c/AGL/AGL-repo/+/29272) | AGL/AGL-repo
- [**\#29280: Add documentation for AGL voice agent**](https://gerrit.automotivelinux.org/gerrit/c/AGL/documentation/+/29280) | AGL/documentation
- [**\#29293: Add pre-trained RASA model**](https://gerrit.automotivelinux.org/gerrit/c/src/rasa-model-agl/+/29293) | src/rasa-model-agl
- [**\#29294: Add pre-trained model for Snips**](https://gerrit.automotivelinux.org/gerrit/c/src/snips-model-agl/+/29294) | src/snips-model-agl
- [**\#29295: Add Snips Inference Module**](https://gerrit.automotivelinux.org/gerrit/c/src/snips-inference-agl/+/29295) | src/snips-inference-agl
- [**\#29296: Add flutter voice assistant app**](https://gerrit.automotivelinux.org/gerrit/c/apps/flutter-speechrecognition-demo/+/29296) | apps/flutter-speechrecognition-demo
- [**\#29338: Update voice agent documentation**](https://gerrit.automotivelinux.org/gerrit/c/AGL/documentation/+/29338) | AGL/documentation
- [**\#29274: Add recipes for voice agent service and app**](https://gerrit.automotivelinux.org/gerrit/c/AGL/meta-agl-devel/+/29274) | AGL/meta-agl-devel
- [**\#29337: Update voice agent service 01**](https://gerrit.automotivelinux.org/gerrit/c/apps/agl-service-voiceagent/+/29337) | apps/agl-service-voiceagent
- [**\#29353: Update voice agent service 02**](https://gerrit.automotivelinux.org/gerrit/c/apps/agl-service-voiceagent/+/29353) | apps/agl-service-voiceagent
- [**\#29354: Update voice agent related recipes**](https://gerrit.automotivelinux.org/gerrit/c/AGL/meta-agl-devel/+/29354) | AGL/meta-agl-devel
- [**\#29400: Add grpc support and improvements to voice agent**](https://gerrit.automotivelinux.org/gerrit/c/apps/agl-service-voiceagent/+/29400) | apps/agl-service-voiceagent
- [**\#29401: Add sample command section to voice assistant app**](https://gerrit.automotivelinux.org/gerrit/c/apps/flutter-speechrecognition-demo/+/29401) | apps/flutter-speechrecognition-demo

If you are familiar with Gerrit, you can also check out all of my commits at one place on [**Gerrit**](https://gerrit.automotivelinux.org/gerrit/q/owner:talhamalik727x@gmail.com).

# Official GSoC Project Link
You can check out my official GSoC project credentials [**here**](https://summerofcode.withgoogle.com/programs/2023/projects/v0K55l17).

# Learning Outcomes
Throughout this GSoC experience, I've gained invaluable insights and experiences that extend far beyond the technical aspects of the project and what I had imagined initially. Some of the new tools, technologies, and key lessons I've learned include:

- **Yocto Project**: I learned how to use the Yocto Project to build a Linux distribution for embedded devices. I also learned how to create custom recipes and layers using the Yocto Project.
- **AGL Eco-System**: I learned how to work with, collaborate on and develop for the AGL eco-system.
- **Flutter & Dart**: I learned how to use Flutter and Dart to create linux based desktop applications. I also learned how to integrate and test flutter applications in the AGL distribution.
- **GStreamer**: I learned how to use GStreamer to create and manage audio pipelines and about integrating GStreamer with Python.
- **gRPC**: I learned how to use gRPC to create a client-server based voice agent service using Dart and Python.
- **Problem Solving**: Tackling unexpected challenges and debugging issues taught me to persevere in the face of adversity and adapt to evolving project requirements.
- **Open Source Culture**: Immersing myself in open-source culture has shown me the importance of giving back to the community, sharing knowledge, and promoting transparency.
- **Continuous Learning**: GSoC has instilled in me the habit of continuous learning, as I encountered new technologies and coding practices throughout the program.

# What's Next?
As I reflect on the journey of developing the Voice Assistant for AGL during GSoC, I am eager to continue my contributions and further refine the project. 

- I am committed to the continuous improvement of the Voice Assistant. This includes addressing any sort of feedbacks, fixing bugs, and staying current with advancements in natural language processing and speech recognition technologies.
- I plan to remain engaged with the AGL community and help any new contributors who are interested in the project. 

# Acknowledgments:
In closing, I would like to extend my heartfelt gratitude to the following individuals who played pivotal roles in my GSoC journey:

- [**Jan-Simon Möller**](mailto:jsmoeller@linuxfoundation.org): Your unwavering guidance, expertise, and encouragement were instrumental in my project's success.
- [**Scott Murray**](mailto:scott.murray@konsulko.com): Your constant support and invaluable insights enriched my learning experience.
- [**Walt Miner**](mailto:wminer@linuxfoundation.org): Your mentorship and feedback were invaluable to my project's success.
- [**Marius Vlad**](mailto:marius.vlad@collabora.com): I appreciate your mentorship and the feedback you provided during this journey.

I am sincerely appreciative of the knowledge and opportunities you have provided, and I look forward to continuing our collaboration in the open-source community. Additionally, I'd like to thank the wider open-source community for their continuous inspiration and collaboration.

This is the final report of my Google Summer of Code 2023 project with The Linux Foundation on Automotive Grade Linux (AGL). In this report, I will be discussing the work I have done in the past 6 months, the challenges I faced, and the things I learned.

GSoC 2023 Final Report: Voice Assistant for Automotive Grade Linux (AGL)


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
This week's activities were primarily focused on project development, documentation, and preparing for the final GSoC report. Notable achievements include the submission of updates to the voice agent service in Gerrit, making necessary improvements to the project's documentation on Gerrit by incorporating internal Gerrit links for enhanced navigation, and the initiation of the first draft of the final GSoC report.

# Tasks completed
- **Voice Agent Service Updates in Gerrit**: Successfully pushed updates to the voice agent service in Gerrit, contributing to the project's development and codebase. These updates play a crucial role in enhancing the functionality and reliability of the voice agent service.
- **Documentation Enhancement on Gerrit**: Improved the project's documentation on Gerrit by incorporating internal Gerrit links. This enhancement optimizes the documentation's usability and ensures a smoother navigation experience for project stakeholders and contributors.
- **First Draft of Final GSoC Report**: Commenced the creation of the first draft of the final GSoC report. This report will provide a comprehensive overview of the project's progress, achievements, and outcomes, serving as a valuable reference for both project assessment and knowledge dissemination.

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, the following task(s) will be undertaken:
- **Finalize GSoC Report**: Complete and publish the final report for my GSoC project.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the twenty second week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 22 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
This week was marked by substantial progress in code contributions to the project. Key accomplishments included pushing the RASA pre-trained model, enhancing and submitting the Snips pre-trained model, pushing the Snips library source code, and submitting the Voice Assistant Flutter app to Gerrit. These contributions significantly advance the project's development and its capabilities for natural language understanding, dialogue management, and voice assistant functionality.

# Tasks completed
- **RASA Pre-Trained Model Submission**: Successfully pushed the RASA pre-trained model to Gerrit. This contribution adds a valuable resource for natural language understanding and improves the project's ability to handle user inputs effectively.
- **Enhanced Snips Pre-Trained Model Submission**: Improved and submitted the Snips pre-trained model to Gerrit. This enhancement ensures the model's reliability and functionality, which is crucial for accurate intent recognition and dialogue management within the project.
- **Snips Library Source Code Submission**: Pushed the source code of the Snips library to Gerrit. This contribution provides access to the core functionality of the Snips platform, enhancing the project's capabilities in voice assistant services.
- **Voice Assistant Flutter App Submission**: Submitted the Voice Assistant Flutter app to Gerrit, contributing to the project's user interface and overall user experience. This app is instrumental in the practical application of the project.

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, the following tasks will be undertaken:
- **Final Gerrit Push**: Push updates and improvements to Gerrit for review and integration.
- **Complete Documentation**: Finalize the documentation.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the twenty first week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 21 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
This week saw notable progress in the project, with a focus on code contributions and documentation. Key accomplishments included the successful submission of the voice agent service and Flutter app recipes to Gerrit, the initial submission of project documentation to the same platform, and an important update to the `snips-nlu-utils` recipe. These achievements are instrumental in advancing the development, collaboration, and reliability of the project.

# Tasks completed
- **Code Contributions to Gerrit**: Contributed to the project's development by pushing the voice agent service and Flutter app recipes to Gerrit. This step is pivotal for collaborative code review, integration, and the ongoing evolution of the project.
- **Documentation Submission**: Successfully submitted the first version of the project documentation to Gerrit. This documentation serves as a valuable resource for project stakeholders, offering insights into the project's architecture, setup procedures, and usage guidelines. It plays a crucial role in ensuring the project's accessibility and usability.
- **`snips-nlu-utils` Recipe Update**: Made an important update to the snips-nlu-utils recipe, modifying it to download from official sources instead of a personal repository. This change ensures that the project is aligned with best practices and maintains the integrity and reliability of its dependencies.

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, the following tasks will be undertaken:
- **Continue Gerrit Push**: Continue to push updates and improvements to Gerrit for review and integration.
- **Documentation Progress**: Continue improving the documentation efforts.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the twentieth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 20 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
This week marked significant progress in the project, focusing on resolving technical issues and advancing the development and documentation efforts. Notable accomplishments include the successful elimination of lag in the WakeWord recognition within the voice agent service, addressing Scipy/OpenBlas linking issues and pushing the fix to Gerrit, and contributing essential components by pushing RASA framework recipes and dependencies to Gerrit. Additionally, significant strides were made in documenting the project, with the completion of the first draft.

# Tasks completed
- **WakeWord Recognition Lag Fix**: Successfully addressed and resolved lag issues associated with WakeWord recognition in the voice agent service. This improvement enhances the responsiveness and overall performance of the voice agent.
- **Scipy/OpenBlas Linking Issue Fix (Pushed to Gerrit)**: Identified and rectified the linking issues related to Scipy and OpenBlas, ensuring proper integration of these critical libraries into the project. The fix was pushed to Gerrit for review and integration.
- **RASA Framework Contributions (Pushed to Gerrit)**: Contributed to the project's development by pushing RASA framework recipes and essential dependencies to Gerrit. This step advances the incorporation of RASA for natural language understanding and dialogue management.
- **Project Documentation**: Made significant progress in documenting the project, with the completion of the first draft. The documentation serves as a vital resource for project stakeholders, providing insights into the project's architecture, setup procedures, and usage guidelines.

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, the following tasks will be undertaken:
- **Continue Gerrit Push**: Continue to push updates and improvements to Gerrit for review and integration.
- **Documentation Progress**: Continue improving the documentation efforts.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the nineteenth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 19 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, significant progress was made in addressing technical challenges and enhancing various aspects of the project. Key achievements include resolving linking issues related to Numpy, Scipy, and OpenBlas, improving the Intent to VSS mapper in the voice agent service, enhancing the Kuksa Client interface, and actively conducting testing on the target AGL image. However, some microphone-related lag issues were encountered during testing. Additionally, the Snips Intent Engine recipes were pushed to Gerrit, advancing the project's development.

# Tasks completed
- **Linking Issues Resolution**: Successfully resolved the Numpy, Scipy, and OpenBlas linking issues, eliminating impediments to the project's functionality and ensuring smooth integration of these libraries. Details about this issue and how it was resolved can be found at this [**Jira Ticket**](https://jira.automotivelinux.org/browse/SPEC-4925).
- **Intent to VSS Mapper Enhancement**: Significantly improved the Intent to VSS (Vehicle Signal Specification) mapper within the voice agent service. This enhancement contributes to more precise and efficient intent recognition and execution.
- **Kuksa Client Interface Enhancement**: Enhanced the Kuksa Client interface within the voice agent service, refining the service's ability to interact with Kuksa for accessing vehicle data and executing commands.
- **Target AGL Testing**: Conducted thorough testing on the target AGL image, actively validating the project's performance in the intended environment. However, encountered microphone-related lag issues, which require further investigation and resolution.
- **Gerrit Contribution**: Pushed Snips Intent Engine recipes to Gerrit, facilitating collaborative development and code review, and advancing the integration of this critical component into the project.

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, the following tasks will be undertaken:
- **Continue Gerrit Push**: Continue to push updates and improvements to Gerrit for review and integration.
- **Documentation Progress**: Continue the documentation efforts, aiming to provide a comprehensive resource that outlines the project's technical aspects, setup procedures, and usage guidelines.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the eighteenth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 18 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, significant progress was made in both the Voice Assistant Flutter App and the Voice Agent gRPC Service. The focus was on resolving compilation errors in the Flutter app, adapting UI components to accommodate limitations in the target AGL Flutter SDK, and conducting initial testing on the AGL target platform. Additionally, for the Voice Agent gRPC Service, comments from the Gerrit code review were addressed, and various refinements were made to enhance the service's performance and reliability.

# Tasks completed
- **Voice Assistant Flutter App**
    - **Compilation Error Resolution**: Successfully resolved compilation errors encountered during the bitbake compilation process. It turned out to be a recipe naming issue. 
        ```bash
        # I was naming my recipe as follows
        agl-flutter-voiceassistant_%.bb
        # Notice the '_%', it was causing issues.
        # Removing it as follows resolved the build errors
        agl-flutter-voiceassistant.bb
        ``` 
    - **UI Component Adaptation**: Adapted the SegmentedButtons component to ToggleButton due to limitations in the target AGL Flutter SDK, ensuring seamless UI functionality in the AGL environment.
    - **Basic Testing on AGL**: Conducted preliminary testing of the Flutter app on the target AGL platform, yielding positive results and confirming the app's operability in the intended environment.

- **Voice Agent gRPC Service**
    - **Code Review Response**: Addressed comments and feedback received during the Gerrit code review process, incorporating necessary changes to align the service with project standards and best practices.
    - **Optimizations and Bug Fixes**: Undertook minor adjustments, optimizations, and bug fixes within the Voice Agent gRPC Service to enhance its overall performance and reliability.


# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, the following tasks will be undertaken:
- **Continue Testing and Gerrit Push**: Proceed with comprehensive testing of the project, ensuring its compatibility with the target AGL environment. Additionally, push updates and improvements to Gerrit for review and integration.
- **Documentation Progress**: Continue the documentation efforts, aiming to provide a comprehensive resource that outlines the project's technical aspects, setup procedures, and usage guidelines.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the seventeenth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 17 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
In the past week, substantial progress was made in both the Voice Assistant Flutter App and the Voice Agent gRPC Service. Key achievements include successful testing of the app's basic UI on the AGL target image and executing mapped VSS signals with the kuksa_client in the Voice Agent gRPC Service. However, challenges were encountered during the Flutter app's recipe build, requiring resolution. The documentation of the project also continued to evolve, and the voiceagent-service was uploaded to Gerrit for review.

# Tasks completed
- **Voice Assistant Flutter App**
    - Conducted testing of the basic App UI on the target AGL image, validating the app's functionality in the target environment.
    - Tested the Dart client's integration with gRPC on the AGL target image. Encountered a recipe build error related to heap space exhaustion, as evidenced by the following error message (cropped):
        ```shell
        do_archive_pub_cache: Error executing a python function in exec_func_python() autogenerated:
        === Crash occured when compiling dart:async__SuspendState@4048458__handleException@4048458 in unoptimized JIT mode in unknown pass
        === Flow Graph not available
        Aborted (core dumped)
        Exhausted heap space, trying to allocate 92672 bytes.
        ../../runtime/vm/virtual_memory_posix.cc: 557: error: mprotect error: 12 (Cannot allocate memory)
        ```

- **Voice Agent gRPC Service**
    - Successfully executed the mapped VSS signals using the kuksa_client. Following signals were included:
        - "Vehicle.Cabin.Infotainment.Media.Volume" 
        - "Vehicle.Cabin.HVAC.Station.Row1.Left.FanSpeed" 
        - "Vehicle.Cabin.HVAC.Station.Row1.Right.FanSpeed" 
        - "Vehicle.Cabin.HVAC.Station.Row1.Left.Temperature" 
        - "Vehicle.Cabin.HVAC.Station.Row1.Right.Temperature."

- **Documentation Continuation**: Continued the work on documenting the entire project, ensuring comprehensive coverage of its architecture, usage, and configuration.
- **Gerrit Upload**: Uploaded the voiceagent-service to Gerrit for review.


# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, the following tasks will be undertaken:
- **Resolve Flutter App Recipe Build Issues**: Address and resolve the bitbake recipe build issues encountered during the Flutter app's integration with the AGL target image. This is crucial for ensuring the app's compatibility and functionality in the target environment.
- **Documentation Progress**: Continue the documentation efforts, aiming to provide a comprehensive resource that outlines the project's technical aspects, setup procedures, and usage guidelines.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the sixteenth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 16 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
This week marked significant progress in the Voice Assistant Flutter App and Voice Agent gRPC Service. The Flutter app now includes a Dart client for streamlined communication with the gRPC server, improved UI elements for enhanced user experience, and robust error handling. Meanwhile, the Voice Agent gRPC Service saw notable advancements in GStreamer pipeline management, the addition of crucial RPC methods, and optimizations for improved performance. Project documentation has also been initiated. The next steps involve rigorous testing on the AGL target machine (x86) and continued documentation efforts to ensure project success.

# Tasks completed
- **Voice Assistant Flutter App**
    - **Dart Client Development**: Developed a Dart client within the Flutter app to facilitate communication with the gRPC server. This client ensures efficient data exchange between the app and the service.
    - **Integration with gRPC Server**: Successfully integrated the Flutter app with the gRPC server, establishing a robust connection for seamless voice assistant functionality.
    - **UI Enhancement**: Improved and enhanced the user interface for both "manual" and "wake word" modes. This enhancement aims to provide a more user-friendly experience.
    - **Configuration Handling**: Implemented a configuration handler capable of loading gRPC configuration parameters (e.g., port, host) from a config.json file, enhancing configurability.
    - **Error Handling**: Introduced robust error handling mechanisms throughout the Flutter app to handle various scenarios gracefully. The app now verifies the availability of the gRPC server before proceeding.

- **Voice Agent gRPC Service**
    - **GStreamer Pipeline Improvements**: Enhanced the creation and deallocation methods of the GStreamer pipeline in the Voice Agent gRPC Service. These improvements contribute to more efficient resource management.
    - **Service Restructuring**: Restructured the service to incorporate additional RPC methods, such as ServiceStatus (for checking service availability) and ExecuteCommand (for executing intents using Kuksa), expanding the service's functionality.
    - **Optimizations and Bug Fixes**: Performed optimizations on the overall service, addressing identified bugs to ensure its reliability and performance.

# Flutter App UI
![Flutter App UI 1](https://i.ibb.co/x845yvz/agl-flutter-app-1.png)

![Flutter App UI 2](https://i.ibb.co/J3dY3x9/agl-flutter-app-2.png)

![Flutter App UI 3](https://i.ibb.co/r4dHJjH/agl-flutter-app-3.png)

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, the following tasks will be undertaken:

- **Testing on AGL Target Machine (x86)**: Conduct thorough testing of the Flutter app on the AGL target machine (x86). Consider exploring Flutter workspace automation further to streamline the testing process and ensure compatibility.
- **Documentation Continuation**: Continue working on the comprehensive documentation of the project. This documentation will serve as a valuable resource for understanding the project's architecture, usage, and configuration.

These next steps align with the project's goals and will contribute to its successful completion.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the fifteenth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 15 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
This week's progress was marked by significant achievements in various aspects of the project. Notably, I successfully designed and developed a mechanism to map intents with Kuksa Vehicle Signal Specification (VSS). Additionally, a new Flutter application was created to seamlessly interact with our voice agent service. Furthermore, I accomplished the bundling and packaging of the voice agent service as a Python package, complete with the ability to function as a background service.

# Tasks completed
- Designed and developed a mechanism to map intents with Kuksa VSS. This mechanism forms the foundation for translating user intents into actionable signals within the vehicle, enhancing the overall user experience.
- Created a new Flutter app specifically tailored to work seamlessly with our voice agent service. This application serves as a critical interface between users and the voice agent service, enabling voice command input and providing a responsive user interface.
- Bundled and packaged the voice agent service as a Python package with the capability to run as a background service. This packaging ensures the service's stability and its ability to function reliably, even in resource-constrained environments.

# Flutter App UI
![Flutter App UI 1](https://i.ibb.co/KKqfmh0/agl-assistant-app.png)

![Flutter App UI 2](https://i.ibb.co/YkPGNyd/agl-fultter-app-image-2.png)

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the coming week, my focus will be on the following tasks:
- Add comprehensive documentation for the intent engines and voice agent service. Thorough documentation is essential for users and developers to understand and effectively utilize these components.
- Integrate and test the newly created Flutter app with the voice agent service on the AGL target machine (x86). This testing phase will ensure that the entire system functions seamlessly and reliably in a real-world environment, bringing us closer to our project's goals.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the fourteenth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 14 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, I achieved significant progress in enhancing the capabilities of the Voice Agent gRPC Service. The main accomplishments included a comprehensive overhaul of the previous GRPC server to establish the new Voice Agent gRPC Service, equipped with various advanced functionalities. Additionally, successful integration of the Python Kuksa client into the voice agent service marked another milestone achieved.

# Tasks completed
- Comprehensive overhaul of the previous GRPC server to establish the new Voice Agent gRPC Service with enhanced capabilities:
    - **Wake Word Mode**: The service now waits for a designated wake word. Upon detection, it promptly signals the client to initiate command execution. This enables a seamless and intuitive user experience.
    - **Auto Mode**: A sophisticated custom Voice Activity Detection (VAD) system was implemented, enabling the service to perform continuous recording and processing of user commands exclusively during the user's speech.
    - **Manual Mode**: A manual recording mode was incorporated into the service's functionality. It records commands based on client cues "Start" and "Stop," providing users with fine-tuned control over the execution of voice commands.
- Successfully integrated the Python Kuksa client into the voice agent service. The integration of the Kuksa client into the voice agent service will allow us to interface with various vehicle-related functionalities.

You can find the updated Voice Agent gRPC Service [**here**](https://github.com/malik727/voicerecognizer-gst-grpc/tree/master/voice_agent_service).

# Tasks leftover
No tasks were leftover this week.

# Next steps
- Utilize the Python Kuksa client interfaces to execute the following intents by implementing mapping interfaces:
    - Volume Controls: Implement functionality within the voice agent service to control audio volume seamlessly using voice commands. This will contribute to a more user-centric and hands-free driving experience.
    - HVAC Controls: Leverage the Kuksa Python client interfaces to enable users to control the vehicle's HVAC system through voice commands. This feature adds convenience and comfort to the driving experience.
- Add comprehensive documentation for the new voice agent service: Develop detailed and user-friendly documentation that explains the setup, configuration, and usage of the enhanced voice agent service. Clear documentation will facilitate seamless adoption and integration of the service by both developers and end-users.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the thirteenth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 13 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, I made substantial progress in enhancing the capabilities of the existing GRPC server by successfully integrating Snips and RASA. Additionally, I extended the RASA and Snips models to incorporate volume control functionality, enhancing the versatility of the voice interaction system.

# Tasks completed
I successfully completed the following tasks over the course of this week:
- Integrated both Snips and RASA seamlessly into the pre-existing GRPC server infrastructure. This integration lays the foundation for a unified and efficient voice interaction system.
- Enhanced the RASA and Snips models by introducing support for volume controls by adding the necessary intents and entities to the models.


# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, I have outlined the following tasks to be completed:
- **Integrate Kuksa**: I plan to integrate Kuksa into the existing GRPC server setup. This integration will enable the execution of intents through Kuksa, expanding the range of functionalities the voice interaction system can offer.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the twelveth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 12 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, I made significant progress towards integrating Snips and RASA into the `meta-offline-voice-agent` project. I focused on updating documentation, committing essential code changes, and creating interfaces for seamless integration.

# Tasks completed
I successfully completed the following tasks over the course of this week:

- Updated the documentation pertaining to Snips & RASA within the meta-offline-voice-agent [**README file**](https://github.com/malik727/meta-agl-devel/blob/master/meta-offline-voice-agent/README.md). This ensures that developers have clear and up-to-date instructions for incorporating these components.
- Implemented discrete commits for all Snips & RASA recipes within the cloned [**meta-agl-devel**](https://github.com/malik727/meta-agl-devel) repository on my GitHub. 
- Extended the existing [**GRPC server**](https://github.com/malik727/voicerecognizer-gst-grpc) by developing interfaces that facilitate the integration of Snips and RASA.

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, I have outlined the following tasks to be completed:

- **Continue GRPC Server Extension**: I will persist in expanding the capabilities of the existing GRPC server. This will involve refining the interfaces created for Snips and RASA integration, optimizing code structure, and ensuring efficient communication between the different components.
- **Exploration of Kuska Docs**: I plan to delve into the Kuska documentation to gain a comprehensive understanding of executing user intents. 

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the eleventh week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 11 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
This week was marked by substantial progress as I focused on enhancing the NLU system's capabilities. Notably, I successfully crafted a custom dataset and trained an NLU model on RASA, covering a range of intents critical to the project. Moreover, I established a recipe for the RASA model, ensuring its seamless local integration into the target machine.

# Tasks completed
- Crafted a custom dataset and trained an NLU model on RASA, encompassing essential intents like:
    - Dashboard Turn Signal 
    - HVAC Temperature 
    - HVAC Fan Speed  
    - Media Player (Play, Pause, Prev, Next)
    - Navigation (To/Stop)
- Created a RASA model recipe, enabling the local deployment of the model on the target machine. This step streamlines the integration process and promotes efficient functionality.

The above new recipes have been incorporated into the existing [**meta-offline-voice-agent**](https://github.com/malik727/meta-offline-voice-agent). You can find the RASA dataset and model files [**here**](https://github.com/malik727/rasa-model-agl).

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, I have outlined the following tasks to be completed:
- Upload the RASA work to Gerrit.
- Extend the existing [**GRPC Server**](https://github.com/amanarora9848/voicerecognizer-gst-grpc/python-server) to effectively parse and execute intents from commands.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the tenth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 10 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, I made substantial progress in my GSoC project, focusing on optimizing the Natural Language Understanding (NLU) system. The key achievements for this week include successfully building Tensorflow by cloning it from the upstream master branch of OE and ensuring all missing dependencies were resolved. Additionally, I conducted rigorous testing of RASA on the target system and efficiently addressed any missing runtime dependencies, ensuring smooth functionality.

# Tasks completed
- Completed the build of [Tensorflow](https://layers.openembedded.org/layerindex/branch/master/layer/meta-tensorflow/) by cloning it from the upstream master branch of OE and resolving any missing dependencies. This step is crucial in preparing the NLU system for advanced model training and inference.
- Conducted testing of RASA on the target system and addressed any missing runtime dependencies. Ensuring that RASA works seamlessly on the target system is a critical milestone towards achieving a reliable and efficient NLU system.

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, I have outlined the following tasks to be completed:
- **Craft dataset and train an NLU model on RASA**: This step will involve preparing a comprehensive dataset and using it to train the NLU model, which will significantly improve the system's understanding and responsiveness to user queries.
- **Start working on the command execution**: Developing the capability to execute commands based on user intent will bring practical utility to the NLU system, making it more versatile and user-friendly.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the nineth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 09 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, I made substantial progress in my GSoC project, focusing on enhancing the Natural Language Understanding (NLU) system. I successfully completed the build of RASA by adding recipes for all its dependencies. Additionally, I optimized the existing NLU model to improve its inference capabilities, enhancing the system's ability to understand and respond to user queries more accurately.

# Tasks completed
- Successfully built RASA by incorporating recipes for all its dependencies. The seamless build process of RASA paves the way for efficient model training and deployment. Following new recipes were added:
    - python3-aio-pika_8.2.3
    - python3-aiogram_2.25.1
    - python3-aiormq_6.4.2
    - python3-jsonpickle_3.0.1
    - python3-magic-filter_1.0.10
    - python3-pamqp_3.2.1
    - python3-pydantic_1.10.9
    - python3-typing-utils_0.1.0
    - python3-tensorflow (loaded as a seperate layer) ([source](https://github.com/nnstreamer/meta-neural-network))
- Enhanced the performance of the existing NLU model by improving dataset.

The above new recipes have been incorporated into the existing [**meta-offline-voice-agent**](https://github.com/malik727/meta-offline-voice-agent).

# Tasks leftover
No tasks were leftover this week.

# Next steps
In the upcoming week, I have outlined the following tasks to be completed:
- **Test RASA on target**: Verify the successful integration of RASA and its dependencies by conducting comprehensive testing on the target machine.
- **Explore RASA documentation**: Delve into the RASA documentation to gain a deep understanding of the model training process. Familiarize with the necessary steps and best practices to effectively train the NLU model.
- **Integrate Tensorflow into existing layer**: Work on integrating Tensorflow into the existing `meta-offline-voice-agent` layer.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the eighth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 08 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
This week was productive and marked significant progress in my GSoC project. I focused on optimizing the existing Natural Language Understanding (NLU) model by incorporating a navigation intent, which will enhance the system's ability to understand and respond to navigation-related queries. Additionally, I successfully developed a recipe to package the NLU model locally, ensuring seamless integration into the target system. To facilitate collaboration and code review, I uploaded all my current work to Gerrit as a patch set.

# Tasks completed
- Integrated navigation intent into the existing NLU model, this enhanced the capability of the model to understand user queries related to navigation.
- Created a recipe that enables packaging the model locally into the target system. This packaging process streamlines the integration.
- Uploaded all the completed work as a patch set to Gerrit. This allows others to provide feedback, suggest improvements, and maintain a clear record of the project's progress.

The uploaded patch set on Gerrit can be found [here](https://gerrit.automotivelinux.org/gerrit/c/AGL/meta-agl-devel/+/28928).

# Tasks leftover
The following tasks were leftover this week:
- **SciPy** and **OpenBlas** linking errors need to be resolved.

# Next steps
- **Create Software Controllers/Daemons**: To execute commands based on user interactions, I will start working on developing Software Controllers or Daemons. These components will enable the system to perform various actions based on the user's intent, expanding the functionality of the overall system.
- **Complete building of RASA**: As part of the project, I will finalize the building process of RASA, an open-source machine learning framework for dialogue systems. 

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the seventh week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 07 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, significant progress was made in the project, focusing on resolving dependencies and successfully compiling the NLU Intent Engine. The compilation of **Scikit-Learn** along with **SciPy** in a single layer was accomplished after making necessary modifications to address conflicts. However, runtime errors related to undefined symbols in **SciPy** were encountered. After extensive investigation, it was determined that these errors were caused by failed linking of **OpenBlas** libraries with **SciPy**. A hypothesis was formulated and tested by manually specifying the **OpenBlas** library using the **LD_PRELOAD** flag, which proved successful in running the intent engine on the target machine.

# Tasks completed
- Made slight modifications to resolve conflicts and successfully compiled **Scikit-Learn** and **SciPy** in a single layer. The **Lapack** and **OpenBlas** libraries were having conflicts over a single header file. Modified the bbappend of **Lapack** by adding following function:
    ```bash
    do_install:append() {
        rm -f ${D}${includedir}/cblas.h
    }
    ```
- Ensured that all the necessary dependencies for the NLU Intent Engine were properly installed and configured.
- Encountered runtime errors related to undefined symbols in **SciPy** during execution.
- Debugged the issue to identify the root cause, which was found to be the failed linking of **OpenBlas** libraries with **SciPy**.
- Temporarily resolved the **SciPy** linking issue by manually specifying the **OpenBlas** library (_located at: /usr/lib/libopenblas.so.0_) using the **LD_PRELOAD** flag.

# Tasks leftover
The following tasks were leftover this week:
- **SciPy** and **OpenBlas** linking errors need to be resolved.

# Running NLU Intent Engine (Optional)
- Clone the updated [meta-offline-voice-agent](https://github.com/malik727/meta-offline-voice-agent) layer into `meta-agl-devel`.
- Update the `templates/feature/agl-offline-voice-agent/50_local.conf.inc` file with the following lines:
    ```bash
    # enabling fortran
    FORTRAN:forcevariable = ",fortran"
    RUNTIMETARGET:append:pn-gcc-runtime = " libquadmath"
    # bad practice but required to build scikit-learn
    HOSTTOOLS += "gfortran"
    # all the dependencies which get included in the final image
    IMAGE_INSTALL:append = " \
        python3-deprecation \
        python3-packaging \
        python3-num2words \
        python3-pyaml \
        python3-requests \
        python3-future \
        python3-scipy \
        python3-scikit-learn \
        python3-threadpoolctl \
        python3-python-crfsuite \
        python3-tabulate \
        python3-six \
        python3-tqdm \
        python3-sklearn-crfsuite \
        python3-snips-nlu-utils \
        python3-snips-nlu-parsers \
        python3-nlu-inference-agl \
        python3-vosk-api \
        vosk-kaldi \
        vosk vosk-server \
        python3-vosk-websocket-server \
        python3-sounddevice \
        flutter-vosk-demo \
        "
    ```
- Build the final image, preferably `agl-demo-platform`.
- After booting into the target image, execute the following command:
    ```shell
    $ export LD_PRELOAD=/usr/lib/libopenblas.so.0
    ```
- Clone the pre-trained model hosted on GitHub using the following commands:
    ```shell
    $ wget https://github.com/malik727/nlu-model-agl/archive/refs/heads/main.zip
    $ unzip main.zip
    $ cd nlu-model-agl-main
    ```
- Finally, load and test the NLU Intent Engine using the following command pattern:
    ```shell
    $ nlu-inference-agl parse /path/to/model-directory -q "your command here"
    ```
    For example:
    ```shell
    $ nlu-inference-agl parse model -q "roll down driver side window"
    ```
To learn more about the NLU Intent Engine model and the currently supported intents, visit the [repository](https://github.com/malik727/nlu-model-agl).

# Next steps
- **Resolve SciPy and OpenBlas Linking Issues**: Continue investigating and addressing the root cause of the failed linking between **SciPy** and **OpenBlas** libraries.
- **Upload to Gerrit**: Add all the lastest work to Gerrit as a patch set after cleanup of unnecessary files.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the sixth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 06 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
This week was highly productive and successful for my GSoC project. I tackled and resolved the issues related to the "snips-nlu-parsers" and "snips-nlu-utils" recipes (thanks to the help from my mentors), successfully building them. Additionally, I dedicated time to crafting a dataset and training the NLU Engine model for crucial intents such as Dashboard Turn Signal, HVAC Temperature, HVAC Fan Speed, Media Player (Play, Pause, Prev, Next), Navigation (To/Stop), and Car Window Controls. Moreover, I expanded the project's capabilities by adding the "python-crfsuite" and "sklearn-crfsuite" recipes.

# Tasks completed
- Fixed rust's network based issues and built the following recipes successfully:
    - snips-nlu-parsers
    - snips-nlu-utils
- Crafted dataset and trained NLU Engine model for the following intents:
    - Dashboard Turn Signal
    - HVAC Temperature
    - HVAC Fan Speed
    - Media Player
    - Navigation (_requires more work_)
    - Car Window Controls
- Added following new recipes to complete the Snips NLU dependencies:
    - python-crfsuite
    - sklearn-crfsuite

The above fixed and new recipes have been incorporated into the existing [**meta-offline-voice-agent**](https://github.com/malik727/meta-offline-voice-agent). The trained NLU Engine along with its dataset can be found at this [**GitHub repository**](https://github.com/malik727/nlu-model-agl).

# Tasks leftover
The following tasks were leftover this week:
- **SciPy** and **Scikit-Learn** recipe errors need to be resolved. (_requires assistance from mentors_)

# Next steps
Looking ahead to the next week, I have identified the following tasks as my focus:
- **Complete building of RASA**: A crucial component of the project that requires careful attention, configuration, and more dependency recipes.
- **Integrate Snips NLU**: Finalize the integration of the Snips NLU into the **meta-offline-voice-agent** layer, ensuring seamless communication and functionality between the components.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the fifth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 05 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, I made progress in reformatting old recipes to adhere to the style guidelines. I successfully uploaded the recipes to the remote machine for collaboration purposes. Additionally, I set up a **Poky Mickledore** build environment with **meta-oe** and **meta-python** layers to compile rust-based Python recipes. I successfully created two recipes, namely **ffi-utils** and **gazetteer-entity-parser**. However, I encountered some challenges while creating the **snips-nlu-parsers** and **snips-nlu-utils** recipes.

# Tasks completed
- Reformatted old recipes to follow style guidelines.
- Uploaded recipes to the remote machine for collaboration.
- Set up a Poky Mickledore build environment for rust-based Python recipes.
- Created the following recipes:
    - ffi-utils (no longer required)
    - gazetteer-entity-parser (no longer required)

The recipes have been incorporated into the existing [**meta-offline-voice-agent**](https://github.com/malik727/meta-offline-voice-agent) layer.

# Tasks leftover
The following tasks were leftover this week:
- Rust based recipes need to be completed.
- **SciPy** and **Scikit-Learn** recipe errors need to be resolved. (_requires assistance from mentors_)

# Next steps
Moving forward, the following tasks will be my focus for the upcoming week:
- **Resolve rust based recipe errors**: Troubleshoot and debug to ensure successful creation of **snips-nlu-parsers** and **snips-nlu-utils**.
- **Begin building RASA**: A task that will require attention and effort to set up the necessary environment and dependencies.
- **Craft Intents Dataset**: This will lay the foundation for further development and training of NLU intent engine.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
The following resources were utilized during the week:
- [Yocto class to populate SRC_URI with crates](https://docs.yoctoproject.org/ref-manual/classes.html#cargo-update-recipe-crates)
- [Migration guide for Yocto Mickledore](https://docs.yoctoproject.org/dev/migration-guides/release-4.2.html)
- [Creating and adding new layers to Yocto](https://kickstartembedded.com/2022/01/19/yocto-part-5-creating-adding-a-new-layer-to-your-image/)

My contribution details and experiences during the fourth week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 04 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, I made significant progress in my GSoC project. I successfully completed several yocto AGL recipes, expanding the existing lapack recipe and creating new recipes for python3-threadpoolctl, python3-joblib, python3-num2words, and python3-pyaml. These recipes were created successfully without any major issues. However, I did encounter some challenges while working on the recipes for Scipy and Scikit-Learn.

# Tasks completed
I developed following Yocto AGL recipes successfully this week:
- Created a recipe for python3-threadpoolctl.
- Created a recipe for python3-joblib.
- Created a recipe for python3-num2words.
- Created a recipe for python3-pyaml.
- Extended the existing lapack recipe to incorporate additional functionality.

The recipes have been incorporated into the existing **meta-offline-voice-agent**, a copy of which can be found [here](https://github.com/malik727/meta-offline-voice-agent).

# Tasks leftover
The following tasks were leftover this week:
- The SciPy recipe errors need to be resolved.
- After building of **SciPy**, **Scikit-Learn** needs to be built.

# Next steps
In the upcoming week, I have outlined the following tasks that I will be focusing on:
- **Resolve SciPy recipe errors**: During the creation of the Scipy recipe, I encountered RPATH errors during the quality assurance (QA) checks. My next step is to investigate and address these errors to ensure a successful recipe for Scipy. An example of the encountered error is as follows:
    ```shell
    ERROR: python3-scipy-1.5.4-r0 do_package_qa: QA Issue: package python3-scipy contains bad RPATH /home/malik/AGL/build-master/tmp/work/corei7-64-agl-linux/python3-scipy/1.5.4-r0/recipe-sysroot/usr/lib in file /home/malik/AGL/build-master/tmp/work/corei7-64-agl-linux/python3-scipy/1.5.4-r0/packages-split/python3-scipy/usr/lib/python3.10/site-packages/scipy/sparse/linalg/eigen/arpack/_arpack.cpython-310-x86_64-linux-gnu.so
    ... and so on
    ```
- **Complete the remaining rust-based recipes**: Apart from the Scipy recipe, there are other two rust-based recipes that need to be developed. I will prioritize completing these remaining recipes, ensuring their successful integration into the yocto AGL environment.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
The following resources were utilized during the week:
- [Yocto - Writing a new recipe](https://docs.yoctoproject.org/dev/dev-manual/new-recipe.html)
- [Building your own recipes from first principles](https://wiki.yoctoproject.org/wiki/Building_your_own_recipes_from_first_principles)

My contribution details and experiences during the third week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 03 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week, significant progress was made in the project. The focus was on updating the Snips NLU library to support **Python >= 3.8** and **NumPy == 1.22.3**. Additionally, the Snips NLU library was refactored into two separate parts: the Intent Engine SDK, which handles preprocessing and training, and the Intent Engine Inference module.

# Tasks completed
- Updated the Snips NLU library to include support for Python >= 3.8 and NumPy == 1.22.3 (NumPy needs to be pre-installed before trying to install the library).
    - The setup.py file was updated to reflect the version change. The current dependency versions are:
    ```python
    required = [
        "deprecation>=2.0,<3.0",
        "enum34>=1.1,<2.0; python_version<'3.4'",
        "funcsigs>=1.0,<2.0; python_version<'3.4'",
        "future>=0.16,<0.18",
        "num2words>=0.5.6,<0.6",
        "pathlib>=1.0,<2.0; python_version<'3.4'",
        "pyaml>=17.0,<20.0",
        "requests>=2.0,<3.0",
        "scipy>=1.0,<2.0",
        "threadpoolctl>=2.0.0; python_version>='3.6'",
        "scikit-learn>=0.20,<0.21; python_version<'3.6'",
        # scikit-learn builds from source and this is the only version that is compatible with Python 3.10 and NumPy 1.22.3
        "scikit-learn==0.22.2.post1; python_version>='3.6'",
        "sklearn-crfsuite>=0.3.6,<0.4",
    ]
    ```
- Refactored the Snips NLU library into two parts:
    - [Intent Engine SDK](https://github.com/malik727/nlu-sdk-agl): This module handles preprocessing and training of intents.
    - [Intent Engine Inference](https://github.com/malik727/nlu-inference-agl): This module handles inference based on the trained intents.
    
    
    Prior to installing the above libraries, it is essential to install the following dependencies, which necessitate a local **Rust** installation and the **setuptools_rust** Python package for compilation:
    ```python
    required = [
        "snips-nlu-parsers",
        "snips-nlu-utils"
    ]
    ```

# Tasks leftover
All tasks for this week were completed successfully with no leftovers.

# Next steps
In the upcoming week, I have outlined the following tasks to further enhance the speech-to-text feature in Automotive Grade Linux and make progress towards achieving the project goals:
- **NLU Engine Recipe Generation**: Create recipes to integrate the Intent Engine SDK and Inference modules into AGL.

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
The following resources were utilized during the week:
- Snips NLU [Docs](https://snips-nlu.readthedocs.io/en/latest/) and [Github Repository](https://github.com/snipsco/snips-nlu).
- [Packaging Python Projects](https://packaging.python.org/en/latest/tutorials/packaging-projects/).

My contribution details and experiences during the second week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 02 Report


# Introduction
Welcome to my weekly report documenting my journey during Google Summer of Code 2023 with the Linux Foundation! In this project, I am working on enhancing the existing speech-to-text feature of Automotive Grade Linux (AGL) by introducing a Natural Language Intent engine and implementing software daemons/controllers to execute the extracted intent. This endeavor aims to significantly improve the user experience and functionality of the speech-to-text feature in automotive environments. Throughout this report, I will share my progress, challenges faced, and achievements made as I contribute to the development of AGL and pave the way for more intuitive and intelligent voice interactions in automobiles.

# Summary of the week
During this week of Google Summer of Code, my primary focus was on setting up and building the AGL master branch repository locally and on a virtual machine. Along with that I made use of the [AGL/meta-agl-devel+28928](https://gerrit.automotivelinux.org/gerrit/c/AGL/meta-agl-devel/+/28928) commit for the integration of "meta-offline-voice-agent" layer.

# Tasks completed
During this week, I made significant progress towards achieving the goals and objectives of my project. Here are all the tasks completed as part of my weekly progress:
- **Repository Setup**: Configured and cloned the AGL master branch repository on my local development environment and on the remote virtual machine.
    - Setup build environment and download repo tool.   
        ```shell
        $ export AGL_TOP=$HOME/AGL
        $ echo 'export AGL_TOP=$HOME/AGL' >> $HOME/.bashrc
        $ mkdir -p $AGL_TOP
        $ mkdir -p $HOME/bin
        $ export PATH=$HOME/bin:$PATH
        $ echo 'export PATH=$HOME/bin:$PATH' >> $HOME/.bashrc
        $ curl https://storage.googleapis.com/git-repo-downloads/repo > $HOME/bin/repo
        $ chmod a+x $HOME/bin/repo
        ```
    - Clone the master branch of AGL.
        ```shell
        $ cd $AGL_TOP
        $ mkdir master
        $ cd master
        $ repo init -u https://gerrit.automotivelinux.org/gerrit/AGL/AGL-repo
        $ repo sync
        $ cd ..
        ```
- **Integration of "meta-offline-voice-agent"**: Utilized the "meta-offline-voice-agent" layer and made necessary modifications and improvements to ensure seamless integration.
    - (Optional) Enable ptests for this layer by adding following two lines into the meta-agl-devel/templates/feature/agl-offline-voice-agent/50_local.conf.inc file.
        ```bash
        DISTRO_FEATURES:append = " ptest"
        EXTRA_IMAGE_FEATURES += "ptest-pkgs"
        ```
- **Build Process**: Built the AGL master branch using the established repository, ensuring a successful build process.
    - Initiate the build process.
        ```shell
        $ source master/meta-agl/scripts/aglsetup.sh -m qemux86-64 -b build-master agl-demo agl-devel agl-offline-voice-agent
        $ source agl-init-build-env
        $ bitbake agl-demo-platform
        ```
    - Now sit back and relax as it can take anywhere from few hours to days for the build to finish depending upon your hardware. (_It took 18-20 hours on Intel i7-8565u 8th Gen CPU with 16 GB of RAM_)


# Tasks leftover
All tasks for this week were completed successfully with no leftovers.

# Next steps
In the upcoming week, I have outlined the following tasks to further enhance the speech-to-text feature in Automotive Grade Linux and make progress towards achieving the project goals:
- **Updating Snips NLU Library**: Update the existing Snips NLU library to support the latest versions of Python. This involves addressing any deprecated or incompatible code, ensuring compatibility with the required Python version.
- **Library Refactoring**: Refactor the Snips NLU library by dividing it into two distinct parts: data preprocessing and model training plus inference. (this work is subjective)

# Conclusion
Overall, this week was productive, and I am satisfied with the progress made in achieving the goals outlined for the week. I am excited to continue my GSoC journey and further enhance the speech-to-text feature in Automotive Grade Linux.

# Resources
There were no resources found attached to this post.

My contribution details and experiences during the first week of coding period of Google Summer of Code (GSoC) 2023.

GSoC 2023 Journey: Week 01 Report


# Introduction
Hello guys! Whether we've crossed paths before or not, let's get acquainted. I'm Malik Talha, a final-year Computer Science student at National University of Computer & Emerging Sciences (FAST - NUCES) in Islamabad. I am the epitome of a model student, tirelessly dedicating my days to the betterment of society - _Can you even imagine?_ 😂. Well, let's cut to the chase and drop the act, shall we? In reality, I'm just like any other person out there — I indulge in gaming, get lost in the captivating world of anime, and probably spend a bit too much time aimlessly scrolling through the vast abyss of the internet (Gotta make the time wasted on shorts and reels sound cool xD). 

Everything aside, here's a delightful surprise: amidst all that, I've discovered a true passion for designing and developing innovative software applications. This passion has always led me into unknown origins and sometimes situations with no exit (referring to my time doing semester projects at university). Should I let you in on one of those captivating incidents? Well, brace yourself because it might just turn into a delightful time-waster. But hey, it's my personal blog, I am allowed to do whatever I want 😉. (You can skip the following paragraph if you're not interested to know about the incident - _your loss_ ☹️) 

> In my fourth semester at university, I enrolled in the intriguing course "Operating Systems." As part of the curriculum, we were tasked with developing a 2D Ludo game using C++, OpenGL, and OS concepts. In a daring move, I decided to explore the realm of 3D game development, despite lacking experience, a tight deadline, and the project's weight on my final grade. Undeterred, I dove headfirst into learning about game development, graphics pipelines, shaders, and 2D/3D rendering. Alongside my studies, I crafted the foundations of my own "Game Engine" to programmatically render graphics and load 3D models. However, as the deadline loomed just three days away, panic struck. I realized I hadn't actually developed the Ludo game itself—a critical oversight! With time slipping away, I made the tough decision to pivot back to a 2D version and meet the deadline. Fortunately, my group mate came to my rescue, contributing to 50% of the game's development just in time. This experience taught me valuable lessons about balancing ambition with practicality, adapting under pressure, and the power of teamwork. On the bright side I was able to develop a bare minimum game engine graphics rendering pipeline which you can find on my [GitHub](https://github.com/malik727/basic-game-engine).

Exploring the depths of the unknown has always been a catalyst for my growth and expertise. Continuing my passion for software development, I decided to apply for [Google Summer of Code 2023](https://summerofcode.withgoogle.com/). This prestigious and highly competetive program by Google offers students and new developers a chance to enhance their programming skills and engage with open-source communities globally. I'm delighted to share that I've been selected 🥳 for The Linux Foundation's Automotive Grade Linux (AGL) project—an exceptional opportunity to contribute to a standardized and customizable software platform for automotive applications. Join me on this exciting journey as I collaborate with the talented developers at Linux Foundation! I will be working under the mentorship of Linux Foundation members: Jan-Simon Möller (Release Manager) and Walt Miner (Community Manager). In the following section I'll dive deeper into my selection process.

# Selection Process
Every March, GSoC reveals the Open Source organizations participating on their website, offering project details and seeking developers. Interested individuals submit detailed proposals outlining their project completion plans before the deadline which is usually first week of April. Mentors from participating organizations evaluate the proposals, and the winners are announced in May. The coding period begines after the winners are announced. 

In my application process, the organization I applied to (The Linux Foundation) introduced a filtering quiz. It required setting up Automotive Grade Linux (AGL) OS locally, developing and publishing a basic app within the AGL environment. The quiz proved challenging, but with the invaluable guidance from the Linux Foundation, I successfully completed it within the given timeframe. Based on my quiz performance and the quality of my proposal I got selected into GSoC 2023.

# Community Bonding
Following my selection into GSoC, the community bonding period commenced in May. This phase allows GSoC contributors to establish connections with their mentors and familiarize themselves with the organization. Currently, I am engaging in separate weekly meetings with my mentors and the AGL community to deepen my understanding of their expertise and grasp the development workflow of AGL. The AGL community comprises of exceptional individuals from whom I anticipate valuable learning experiences. It is a privilege to be a part of this remarkable community. Once the bonding period concludes, I will start work on my designated project.

# My Project for GSoC 2023
Introductions and details about GSoC aside, let's discuss about what project I will be working on. Before I dive into my project details further, I'll give a brief description about AGL:
> Automotive Grade Linux (AGL) is an open-source software platform specifically designed for automotive applications. It provides a standardized and customizable solution for connected cars and infotainment systems. AGL offers a comprehensive set of features, including a Linux-based operating system, middleware, and application framework. It enables automotive manufacturers and developers to collaborate, accelerate innovation, and create secure and reliable software for vehicles. AGL fosters an ecosystem of industry stakeholders, promoting interoperability and driving the adoption of open-source software in the automotive industry.

## Overview
My project is to enhance the existing speech recognition engine of AGL by developing and integrating a natural language intent engine that understands and executes users' voice commands.

## Goals and Objectives
The goal of my project is to develop a car's voice assistant which can understand voice commands and take necessary action based on them (e.g. turn down the volume, increase AC temperature, play a song, etc.). There are two main objectives of my project:
- **Objective # 01:** Integration and development of Natural Language Understanding Intent Engine with existing Vosk implementation.
- **Objective # 02:** Ability to execute the intent extracted from the NLU engine using some sort of interface that will communicate with the APIs.

## Tools and Technologies
I will be working with the following tools and technologies as part of my project:
- **Linux:** AGL is built on a Linux-based operating system.
- **Yocto Project:** AGL utilizes the Yocto Project build system for creating customized Linux distributions.
- **Bitbake:** Bitbake is a build tool used in the Yocto Project for creating customized Linux distributions.
- **Wayland/Weston:** AGL uses the Wayland protocol and Weston compositor for graphical display and user interface rendering.
- **Flutter:** Flutter is a cross-platform framework for building native mobile and web applications.
- **C/C++:** AGL involves developing software components and applications using C/C++.
- **Python:** The Natural Language Understanding Intent engine will be developed in python.
- **Collaborative Tools:** Communication and collaboration are vital in open-source projects. Tools like mailing lists, IRC (Internet Relay Chat), and project management platforms (e.g., Jira) are commonly used in the AGL community.

## Challenges and Expectations
I am expecting to face following challenges:
- **Natural Language Understanding:** Understanding the intent behind user commands requires robust natural language understanding (NLU) capabilities. NLU involves parsing and extracting relevant information from user inputs, which can be challenging, especially when dealing with complex or ambiguous commands.
- **Integration with Car APIs:** To execute commands such as adjusting car volume, your voice assistant needs to interact with the underlying car APIs and systems. Understanding and integrating with these APIs may require in-depth knowledge of the car's software architecture and protocols.
- **Context Awareness:** A voice assistant should be context-aware to provide relevant and accurate responses. Understanding the context of the user's request, such as the current state of the car or previous interactions, can enhance the user experience but requires careful implementation.
- **Performance and Latency:** Voice assistants should respond quickly and perform tasks efficiently to provide a smooth user experience. Optimizing performance and minimizing latency can be challenging, especially when dealing with resource-constrained embedded systems.

# Conclusion
In conclusion, my journey into the world of GSoC 2023 and Automotive Grade Linux has been nothing short of exhilarating. From the thrill of being selected to the engaging community bonding period, I am ready to embark on this remarkable project. With the support of my mentors and the vibrant AGL community, I am confident that this experience will be filled with learning, growth, and exciting developments. Stay tuned as I dive into the realm of voice assistants, bringing innovation to the automotive landscape. Thank you for joining me on this adventure, and let's make some magic happen together! 

_(P.S. If you're interested about the actual proposal I submitted for GSoC 2023 let me know and I'll share it as a blog post.)_

Follow along as I share my project overview and experiences during the community bonding period of Google Summer of Code (GSoC) 2023. Learn about my selection process, the project I'll be working on, and the exciting milestones ahead.