Springing into AI - Part 7: Observability

-

 Welcome back, and hope you having a wonderful day. In Part 6 of the series, we had a look at our first exciting chat application where we were finally able to get hands dirty and interact with the LLM that was running on our local machine. Before we get our hands more dirty with more topics, we pause a bit to understand an important concept of "Observability".  

        Backend applications like the one we created previously require love from us engineers. This isn't the same type of love that was shared in Titanic by Leo and Kate, but more of a monitoring love that empowers us to know the state of our application from some fine grained metrics that is of interest to us so that we may have a behavioral sense of our running application. A typical example, in a Java application maybe wanting to know the amount of memory used in JVM (Java Virtual Memory), or the number of threads in use, information about garbage collection, etc. When we speak of GenAI applications, tokens form our social currency, and it's usage can ramp up quite quickly providing us a fat bill to pay if we not careful. To avoid such a situation we can do something about it. Someone please picture Darth Vader going like "If only you knew the power of Observability". 

Observability - Theory

    Prometheus and Grafana are the most commonly used tools adopted enterprise wide for monitoring an application's state at a given time. While Prometheus allows us to collect, store and query the metric data provided by the application, Grafana allows us to query, visualize and alert on the data that we may take actions where applicable. In context of our application, the figure below presents an overview of how these components integrate into it. 



  • Spring Actuator: Spring framework provides us actuator endpoints that provide us some key metrics about the state of our backend application, e.g.: health, information. In our application we also enable "prometheus" here. This is of vital important to the whole system, as the exposed endpoint would be used by "Prometheus"  tool  to collect various metrics for further usage. SpringAI offers variety of such metrics from AI perspective such as model parameters, token usage, etc. More about the various exposed metrics can be found here. The image below shows a metrics observed by our application that is gathered by invoking few prompts. 

  • Prometheus: As mentioned above, the metrics exposed on a particular endpoint is scraped by this tool. For our AI application, we would definitely like to know the use of number of tokens (input, output, total) so that we can take actions (limit the context window on model parameter as an example) to prevent cost incurring beyond our pocket can afford. The image shown below is from the user interface Prometheus has where we have searched for a particular metric amongst many other options available as an example for illustration purposes.

  • Grafana: Okay, so Spring actuators expose the metrics to outside world. Prometheus has scraped these metrics, stored and enabled us to query. Wouldn't it be nice if we can visualize these metrics in form of a dashboard that we can visit and gain insight via different display widgets ?, enters "Grafana" into the picture. The image below shows a sample dashboard observed from our application where we can already see some key information. As we use our application more and more, it would update almost real-time based on the scraping interval. 

        Special mention here to Dan Vega who is a Spring advocate developer. His work has helped me in this journey learn through his medium of tutorials, videos and articles. The above dashboard that you see is sourced from his work where he had created a typical prometheus-grafana dashboard setup for us to use. For a more in depth tutorial, you can view his youtube video tutorial. In above dashboard, we can see the amount of invocations we have done using our endpoint to prompt with LLM, the response times, the token usage, response time duration amongst others to help us know how insights  of our application usage. 

Observability - Playground

    Alright, now that we have an understanding of what all tools are, and how they work and expose data to us, let's have a look at how this looks from a code point of view. Most of the code remains same as before, with few enhancements added.  
  • Source Code: Can be accessed here
  • Dependencies:
    • spring-boot-starter-actuator: It enables actuator endpoints that can be scraped by "Prometheus". Through configuration we enable prometheus as one of the endpoint.
    • spring-boot-docker-compose: Out setup makes use of docker-compose file namely "compose.yaml" to setup and run Prometheus and Grafana respectively. As part of the container setup, we also load pre built Grafana dashboard template and a configured Prometheus datasource used by it for data population. Using the mentioned dependency allows us to automatically run these containers at startup instead of us manually managing these containers every time. 
  • Configuration: 
    • Management: Using spring properties we enable certain metrics and setup time capture intervals that would be used by our tools to help our vision be realized.
  • Endpoints:
    • Prometheus Spring Actuator: http://localhost:8080/actuator/prometheus
    • Prometheus Tool: http://localhost:9090/
    • Grafana: http://localhost:3000/
        I encourage you to check the code out and play by running few prompts and see your application's observability in action. This should form the basis and stock standard setup for most of our experiments going forward. In the next section we will look at more advanced prompts covering things as previously promised like chat memory, system prompts and much more. Stay tuned... 



Comments

Post a Comment

Popular posts from this blog

Springing into AI - Part 4: LLM - Machine Setup

-
Image
     Welcome back, wherever you are, hope you having a wonderful day. So far in the series, we had a theoretical look at the eco-system of AI, and then focusing our journey into Generative AI where we now at least know on a surface level of its working and some techniques in how we can best interact with it through effective prompting. Before we start getting our hands dirty into writing our own AI applications, we need to look at options available to us when working with LLM. These can range from using an external provider or using our own local machine as a base for running a LLM. The figure below provides an illustrative example of some options (more may follow in future).      From the figure above, some of the options we have. Cloud Providers:  You can leverage the offering via various cloud providers like AWS, Azure etc. to interact with the foundation models. For example, AWS presents Bedrock that offers the ability for you to select the foundat...
Read more

Springing into AI - Part 1: 1000 feet overview

-
Image
 Artificial Intelligence (AI)....We live in the world where technology trends adapts in one form or another through our needs as humanity evolves. However the field of AI isn't a recent discovery. It has existed for years and recently gained even more visibility through its offering aimed to make our lives easier, depending on how we adopt it.      The first part of the series will focus on providing a high level overview of sorts into the world of AI so we have clarity on its eco-system before we start to get our hands dirty with playground of how we can use it for our custom applications.   Ecosystem The eco system of Artificial Intelligence is composed of various sub-sections. Figure below should provide you a concise summary of its elements.   Let's have a summarized break down. Note: There exists plethora of material on web to dive into every topic, I won't be covering it all, as my grand children will have their children by the time I am done wri...
Read more

Springing into AI - Part 2: Generative AI

-
Image
      Welcome, In this post of the series we continue our journey by looking into the  wonderful world of Generative AI. Specifically focussing our core towards Large Language Model (LLM) that form the backbone of GenAI applications. We will understand on high level their way of working, factors to consider when choosing a foundation model, and certain options available in fine tuning its behaviour.  Large Language Model (LLM)        These are AI models composed of billions and billions of neurons that are pre-trained upto a certain moment in time from wide range of sources, such as internet, publications, research materials, etc . While they are very powerful, their pre-trained nature imposes a limitation as depending on a use case they may not be capable of answering anything beyond its finite knowledge base. In subsequent posts we will cover sections that can help mitigate this based on our use cases. The figure below presents an evolution...
Read more