AWS Partner Network (APN) Blog
How Freshworks developed CloudVerse, an enterprise-grade GenAI platform on AWS
By Sundeep Paluru, Senior Engineering Manager AI/MLOps Platform – Freshworks
By Jayasundar S, Sr. Partner Solution Architect – AWS
By Sriram Sundararajan, Sr. Solution Architect – AWS
 |
| Freshworks |
 |
Generative artificial intelligence (AI) has made it easier for many users to leverage foundation models (FMs) to transform products and experiences across industries. However, organizations looking to operationalize these generative AI capabilities face several challenges including cost efficiency, data privacy concerns, and future-proofing in a rapidly evolving field.
To address these challenges and unlock the full potential of generative AI, Freshworks has developed CloudVerse. CloudVerse AI platform that provides AI as a service (AIaaS), designed to enable adoption across applications such as chatbots,
Freshworks’ cutting-edge AI offering, called Freddy AI, is powered by CloudVerse, a platform component that provides AIaaS. By leveraging CloudVerse’s AIaaS capabilities, various business units within Freshworks can seamlessly integrate AI-driven tools into their operations, delivering personalized experiences and driving efficiency across various touch points without extensive in-house AI resources or expertise.
CloudVerse, developed in collaboration with AWS, enabled Freshworks to accelerate building generative AI applications on AWS. Read on to learn more about how AWS enabled Freshworks to leverage AIaaS offerings and helps them stay ahead of the curve with various services, including Amazon EKS, AWS Lambda, and Amazon Bedrock.
AWS and Freshworks have built multiple integrations to provide customers a better experience in managing and using AWS resources with the power and flexibility of Freshworks solutions. In addition to being a member of the AWS Partner Network (APN) and an AWS Public Sector Partner, Freshworks is a participant in the AWS independent software vendor (ISV) Accelerate program and also offering solutions in AWS Marketplace.
Genesis of CloudVerse
At Freshworks, the vision of democratizing AI within the organization led to the creation of CloudVerse. Inspired by the desire to make AI accessible and usable for all departments, CloudVerse was born with the primary objective of creating a flexible and powerful AI platform that empowers Freshworks teams to experiment and deploy AI solutions with agility and ease.
By utilizing AWS infrastructure and through the efforts of Freshworks’ AI engineers and experts, the company successfully transitioned CloudVerse from concept to production in under three months.At the core of CloudVerse lies an unwavering dedication and belief in the transformative power of AI. By enabling teams to harness the potential of AI, Freshworks aims to enhance its products and services while fostering a culture of continuous learning and innovation.
Unleashing the Power of Prompt Engineering with CloudVerse’s AI-Centric Platform
Playground
At the heart of leveraging large language models (LLMs) effectively lies the art of crafting the right prompts, known as prompt engineering.
CloudVerse features an AI-centric playground with an intuitive user-friendly interface, enabling Freshworks’ internal teams to perform prompt engineering across various commercial and self-hosted open-source models.
Within this innovative platform, employees can define their unique use cases and delve into prompt engineering, simultaneously comparing prompt responses across different LLMs. Responses are meticulously evaluated based on quality, performance, and the number of tokens generated, enabling data-driven decision-making.
Furthermore, CloudVerse has introduced an innovative feature, allowing prompt engineers to seamlessly collaborate and contribute to Freshworks’ prompt engineering community. By accessing prompts through a magic link, engineers can effortlessly share their use cases, fostering a culture of knowledge exchange and accelerating the advancement of prompt engineering practices.
With CloudVerse’s AI-centric platform, Freshworks empowers its teams to unlock the full potential of LLMs, by streamlining the prompt engineering process, facilitating cross-model comparisons, and promoting a collaborative environment for continuous improvement and innovation.
Unified API Interface
CloudVerse offers a unified Application Programming Interface (API), providing Freshworks’ teams with the flexibility to integrate their applications with the platform’s API. Teams can generate a JSON Web Token (JWT) and leverage it to access CloudVerse’s API endpoints, enabling seamless interaction with multiple managed and self-hosted models. This unified API interface serves as a one-stop solution for application integration, empowering employees to harness the power of LLMs within their existing workflows.
Model Hosting Capabilities
CloudVerse’s AI platform utilizes a modern technology stack to facilitate the efficient hosting and deployment of LLMs and other AI models. The platform employs the following key components:
- TensorRT-LLM: CloudVerse utilizes TensorRT-LLM, a high-performance inference library from NVIDIA, to optimize LLMs for reduced latency and improved throughput. This library enables efficient deployment of massive LLMs on GPU-accelerated infrastructure.
- Ray-llm: To simplify the setup and deployment of LLMs on Elastic Kubernetes Service (EKS), CloudVerse integrates Ray-llm, a scalable and distributed computing framework. This framework streamlines the serving of LLMs at scale on EKS clusters.
- Amazon Elastic Kubernetes Service (EKS): CloudVerse’s platform is built on top of Amazon EKS. Amazon EKS provides a robust and scalable infrastructure for hosting and orchestrating containerized AI models and applications.
- JARK Stack: Inspired by the JARK (Jupyter, ArgoCD, Ray, Kubernetes) stack, CloudVerse’s cloud engineering team has developed a self-hosted machine learning (ML) model deployment solution. This solution leverages the power of Kubernetes for managing and scaling AI workloads.
- Best Practices: CloudVerse’s platform incorporates industry best practices, such as autoscaling, fractional GPU utilization, and model multiplexing. These practices ensure optimal resource utilization, cost-efficiency, and high-performance AI model hosting.
By combining cutting-edge AI technologies like TensorRT-LLM and Ray-llm with robust cloud-native infrastructure like Amazon EKS, CloudVerse empowers organizations to seamlessly deploy and scale AI models. It also provides support for massive LLMs, while adhering to best practices for performance, scalability, and cost optimization.
Centralized Governance Capabilities
Freshworks maintains control over its AI initiatives through centralized AI governance, aligning them with strategic objectives while proactively mitigating risks associated with AI adoption. This approach fosters trust and accountability in AI systems, promoting the responsible use of AI technologies that prioritize ethical considerations, human oversight, and societal well-being.
Freshworks has established clear guidelines and processes for AI development, deployment, and monitoring with centralised governance. It enables data protection, mechanisms for identifying and mitigating algorithmic biases, and robust validation procedures to ensure the accuracy, reliability, and safety of AI models.
The centralized AI governance facilitates the establishment of cross-functional teams and advisory boards to oversee AI initiatives. It ensures multidisciplinary expertise and diverse perspectives are incorporated into decision-making processes.
Solution Architecture
CloudVerse has built an AI as a Platform by leveraging Ray, a distributed computing framework, deployed on Amazon EKS. This platform takes advantage of the scalability and reliability of AWS services to provide an efficient and robust infrastructure for running AI and machine learning workloads.
The below diagram shows the high level architecure with LLM used in the environment.
Figure 1 – High Level AIaaS – Architecture
At the core of this platform is Ray, an open-source unified framework designed specifically for scaling AI and Python applications. Ray’s distributed computing capabilities ensure efficient workload distribution across multiple nodes and GPUs, enabling seamless parallelization and scalability of machine learning tasks. The platform is hosted on EKS, a managed Kubernetes service provided by AWS, allowing for containerized deployment and management of applications.
Integration with other AWS services, such as Amazon S3 for storage and AWS Lambda for serverless computing, further enhances the platform’s capabilities, enabling reliable data management and efficient execution of specific tasks.
Ray’s comprehensive set of components and features, including scalable libraries for common machine learning tasks, distributed computing primitives, and integrations with existing tools and infrastructure, make it well-suited for building an AI as a Platform. This platform simplifies the process of scaling jobs for data scientists and machine learning practitioners, while providing a scalable and robust environment for ML platform builders and engineers.
Impacts of CloudVerse
Democratization of AI: CloudVerse aims to make AI accessible to all departments within Freshworks, fostering a culture of innovation and continuous learning. This democratization can lead to a more agile organization where teams can experiment and deploy AI solutions without needing extensive expertise.
Operational Efficiency: By automating repetitive tasks and providing predictive insights, CloudVerse enhances operational efficiency. This can lead to reduced costs and improved productivity across various business functions.
Enhanced Customer Experience: With tools like Freddy AI, businesses can deliver personalized experiences to customers, improving customer satisfaction and loyalty. The integration of AI in customer support can lead to quicker response times and more accurate solutions.
Scalability and Flexibility: The use of AWS infrastructure allows Freshworks to scale its AI capabilities efficiently. This flexibility enables the organization to adapt to changing market demands and technological advancements.
Centralized Governance: The establishment of centralized AI governance ensures that AI initiatives align with strategic objectives while addressing ethical considerations. This can build trust among stakeholders and mitigate risks associated with AI adoption.
Cost Optimization: Early projections indicate that CloudVerse could help Freshworks reduce infrastructure costs by 30% and increase resource utilization by 40%.
The launch of the Cloudverse marked a significant milestone in Freshworks’ journey towards fostering a culture of innovation and collaboration. The platform’s success was undeniable, with its benefits becoming rapidly evident across the company.
Benefits of CloudVerse
Rapid Deployment: The ability to transition from concept to production in under three months demonstrates the platform’s efficiency, allowing Freshworks to quickly capitalize on AI opportunities.
User-Friendly Interface: The AI-centric playground simplifies prompt engineering, making it easier for non-experts to engage with AI technologies and contribute to AI initiatives.
Robust Infrastructure: Built on AWS services like Elastic Kubernetes Service (EKS), CloudVerse provides a reliable and scalable environment for running AI workloads, ensuring high performance and availability.
Integration Capabilities: The Unified API Interface allows seamless integration of AI tools into existing workflows, enhancing the overall functionality of Freshworks’ applications.
Data-Driven Decision Making: The platform’s capabilities for evaluating prompt responses based on quality and performance enable teams to make informed decisions, improving the effectiveness of AI applications.
Future-Proofing: By leveraging cutting-edge technologies and maintaining a focus on best practices, CloudVerse positions Freshworks to stay ahead in the rapidly evolving AI landscape.
Ethical AI Practices: The centralized guardrails and governance structure promotes responsible AI use, ensuring that ethical considerations are integrated into AI development and deployment processes.
Conclusion
In this post, we highlighted Freshworks CloudVerse and how it represents a significant stride in democratizing AI and fostering a culture of innovation within the organization. By leveraging LLMs, prompt engineering, and cloud-native infrastructure, CloudVerse empowers teams to experiment, deploy, and scale AI solutions with ease and efficiency.
Adhering to best practices and a centralized governance approach, powered by the scalability and reliability of AWS, CloudVerse provides an efficient and robust infrastructure for running AI and machine learning workloads. The integration of Ray, a distributed computing framework, further enhances the platform’s capabilities, enabling seamless parallelization and scalability of machine learning tasks.
Overall, CloudVerse represents a significant leap in Freshworks’ AI journey, positioning the company at the forefront of innovation and enabling it to deliver exceptional customer experiences through intelligent solutions.
.
Freshworks – AWS Partner Spotlight
Freshworks an AWS Differentiated partner with vision to deliver modern and innovative AI-enabled customer and employee service solutions that enable companies of all sizes to deliver personalized experiences and increase productivity. Also, Freshworks is an AWS Retail Competency, Small and Medium Business Software Competency and AWS Cloud Operations Competency partner. Freshworks is a participant in the AWS ISV Accelerate program and also offering solutions in AWS Marketplace.