I have implemented a complete backend, different microservices accross the projects, data and annotation pipeline for a pest management project for which Wadhwani AI won Google’s AI impact challenge and got a 2 million grant from Google. I enjoy working with all layers of the stack and have a strong belief in automation. I am more of a generalist with strong backend skills. Prior to this, I was leading engineering team engineering teams across multiple projects in the healthcare and agriculture domains at Wadhwani Institute for Artificial Intelligence which is an independent, non-profit research institute and global hub, developing AI solutions for social good. I have contributed to Apache Airflow and OpenCV CVAT repositories. I am a strong advocate of open-source, and believe in Learning by Doing. Built on Kubernetes, Astronomer makes it easy to run, monitor, and scale Apache Airflow clusters in our cloud or yours. Astronomer is the orchestration platform for the modern data stack. If you are interested in learning more about Kubernetes Executor in Astro, please visit our docs or contact us directly for a demo.I am a Senior Software Engineer at Astronomer, where I work on design and engineering of Apache Airflow (an open source workflow management platform). With this new capability, we are confident that our customers will be able to build more scalable and reliable data pipelines, while reducing the complexity and cost of managing Kubernetes. We are excited to see the impact it will have on their data pipelines and how they leverage Airflow for data processing and orchestration. Kubernetes Executor support is now available in Generally Available for our Hybrid customers and Public Preview for our Hosted customers. Simplified Management: By using Astro, customers can avoid the complexity of managing their own Kubernetes clusters, while still enjoying the benefits of the Kubernetes Executor.Fault-tolerance: With Kubernetes, Airflow tasks will continue to run even if there are problems with other Airflow components and can run 24h+ workloads, ensuring that data pipelines remain reliable and resilient.Resource Management: Kubernetes provides fine-grained control over resources such as CPU and memory, allowing customers to optimize their use of resources and reduce costs while making sure each workload has access to the resources it needs. Scalability: Scale up or down based on demand, making it easy to handle fluctuations in workload and avoid performance issues.The benefits of using Kubernetes Executor in Astro include: Kubernetes Executor in Astro delivers all the benefits of Kubernetes for Airflow while Astronomer manages the clusters themselves, allowing data practitioners to focus on building and running their data pipelines. This makes the Kubernetes Executor an ideal choice for organizations that require high performance and reliability, as well as the ability to handle large volumes of data and compute-intensive workloads. Each Airflow task is run in its own isolated environment, which provides more fine-grained control over resources like CPU and memory. Kubernetes Executor is a popular option for running Airflow tasks in a distributed and scalable manner. If you are using Kubernetes Executor overrides today, that means no more rearchitecting and refactoring pipelines when migrating to Astro! Your dags and tasks will run on Astro with minimal-to-no changes. As of today, we are the only managed Airflow service provider that offers support for Kubernetes Executor. If you are already running Airflow with the Kubernetes Executor, the path to Astro just got a lot easier. With this new capability on Astro, you can now take advantage of the underpinning power of Kubernetes to manage resources and scale your Airflow workloads. We are excited to announce the launch of Kubernetes Executor support in Astro, our managed Airflow service and data orchestration platform.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |