< Passionate about designing and optimizing high-performance, scalable data pipelines and implementing AI-driven solutions. Proficient in Python, SQL, Elasticsearch/OpenSearch, Scrapy, Selenium, Playwright, Crawl4ai, API development, Amazon Athena-Glue, and a broad range of AWS services, with a strong focus on enabling data-driven decision-making. Exploring the LLM and AI tech stacks - Llama.cpp, CrewAI, LangGraph, etc. />
Insights and experiences from my journey in data engineering and software development.
SQL window functions are powerful tools for analytical queries, especially in data-heavy environments. They were created to address limitations in traditional SQL operations, particularly for performing analytical calculations across related rows without collapsing the result set.
It is a very simple SQL concept but heavily used. Usually, we use it with one joining / matching condition, but sometimes there can be multiple conditions depending on the scenar
We need to use field data caching to utilize fast aggregation and sort queries. However, caching too many fields can degrade Elasticsearch performance by reducing available JVM memory.
Do you know why the π£πΏπΌππΆππΆπΌπ»π²π± π§π΅πΏπΌππ΄π΅π½ππ (π πΆπ/π) and ππ’π£π¦ (Input/Output Operations Per Second) are very important stuffs to consider based on requirements?
We may encounter an yellow cluster status of an AWS Elasticsearch cluster. Do we know why dpes the cluster run in yellow state?
-lifecycle-elasticsearch.webp&w=1080&q=75)
An Entire Lifecycle of An Indexing Request (Single & Bulk) in Elasticsearch. When we send a document to be indexed, here is the sequence of events that must happen before our client/service gets a 200 OK or 201 Created response.
