Springing into AI - Part 4: LLM - Machine Setup

Problem 

Generative AI has these "magic" models that take the user request, processes and provides an output. As developers we want to be able to use these LLM models in a flexible way that allows us to use them for experimentation and playground our solutions.

Solution

As the community continues to evolve at a rapid pace, there exist various options at our dispoal that allows us freedom to opt for an approach that works best for your choice of methodology. The figure below provides an illustrative example of some options (more may follow in future). 

    From the figure above, some of the options we have.

• Cloud Providers: AWS, Azure to name a few offer foundation models that we can use in our applications to integrate with. For example, AWS presents Bedrock that offers the ability for you to select the foundation model you want to use in your application. These models we can then interact with via our third party application or should you wish fashion a design with Lambda's. It it is to be noted that these do come with costs, and tokens are the social currency when working with Foundation Models.

• External Providers: Companies like Anthropic, OpenAI also offer the flexibility of using their models on a subscription basis where a developer API key is assigned that one can register for, and then use the different models. It cannot be stressed enough that while this is amazing, there comes cost and tokens are the social currency when working with GenAI applications.

• Containers: With cloud and external providers, we always have to be weary of the cost that comes with using the LLM's. If that is a concern and you find yourself limited, there are container based options from Docker that allows you run these LLM's locally on your machine. The choice of model you chose does comes with resources required to be able to run them as they vary in size and computing power. I tried the two options lsited below:

• Docker Model Runner: Using docker desktop, you can install the docker model plugin and enable it. This plugin allows you a massive benefit of pulling the model images locally onto your machine form docker hub, that you can then choose to interact with. Do note, you would need a juicy up machine to be able to run some of the models, and also the machine maybe constrained by the resource capacity of the container.  

• Rancher Desktop: For development purposes if you don't want to use Docker Desktop, you can use Rancher Desktop to install Docker, along with Kubernetes (should you wish). The software offers wide variety of extensions, the one in particular you would be looking for is "OpenWebUI". This under the hood will install some docker images on the machine. Coupled with Ollama installed on your machine, either separately or through the software itself, it offers you then the ability to choose whatever model you want from it's library and interact with it. Instruction for installations and setup can be found at Rancher Desktop Installation. The instructions are pretty self explanatory to help you setup.

• Software Setup: Instead of going the container or a cloud provider route, you can manually install 
• Claude for Desktop provided from Anthropic enables you to use an interface empowers you to then interact with the foundation model. It also has capability to work with own tool and functional calling. (We will discuss this later in the series when we cross that bridge).  
• Ollama provides a vast library of models at your disposal that you can download on your machine and then use that as a basis for interacting, developing your applications with. In Part 2 of the series we discussed model evaluation which can help you decide some criteria for selecting the model for your experimentation. Depending on the speed of your network and machine power, you may opt for a heavy model or a simple basic model for your playground. But don't worry as Gandhalf said "Even the smallest person can change the course of the future", and a small scale model is no different for you to practice and polish your skills. Once the software is installed you can run various commands, some of which are below: