< I build the data infrastructure behind a Swedish B2B sales intelligence product β pipelines processing 10M+ records daily into a 300M+ document Elasticsearch index, an S3 + Apache Iceberg lakehouse queried through Athena, and search clusters I've re-architected to cut AWS costs 43%. Since 2023 I've been building the AI layer on top: a self-hosted LLM on llama.cpp, MCP servers, LangChain and LangGraph agents, and a RAG proof-of-concept over our Elasticsearch data that lets clients query companies, news, and recruitment in natural language. Python Β· SQL Β· Elasticsearch/OpenSearch Β· AWS Β· Iceberg Β· Athena Β· Glue Β· Scrapy Β· Crawl4ai Β· Playwright Β· FastAPI Β· llama.cpp Β· LangChain Β· LangGraph Β· MCP Β· RAG />
Insights and experiences from my journey in data engineering and software development.
SQL window functions are powerful tools for analytical queries, especially in data-heavy environments. They were created to address limitations in traditional SQL operations, particularly for performing analytical calculations across related rows without collapsing the result set.
It is a very simple SQL concept but heavily used. Usually, we use it with one joining / matching condition, but sometimes there can be multiple conditions depending on the scenar
We need to use field data caching to utilize fast aggregation and sort queries. However, caching too many fields can degrade Elasticsearch performance by reducing available JVM memory.
Do you know why the π£πΏπΌππΆππΆπΌπ»π²π± π§π΅πΏπΌππ΄π΅π½ππ (π πΆπ/π) and ππ’π£π¦ (Input/Output Operations Per Second) are very important stuffs to consider based on requirements?
We may encounter an yellow cluster status of an AWS Elasticsearch cluster. Do we know why dpes the cluster run in yellow state?
-lifecycle-elasticsearch.webp&w=1080&q=75)
An Entire Lifecycle of An Indexing Request (Single & Bulk) in Elasticsearch. When we send a document to be indexed, here is the sequence of events that must happen before our client/service gets a 200 OK or 201 Created response.
Trusted by Industry Leaders
Senior Software Engineer | BigData | NoSql | GraphDB | Machine Learning
January 15, 2024, Ashraful Islam worked with Mehadi in the same data team
From the perspective of a team leader, I had the pleasure of working alongside Md. Mehadi Hasan, and he consistently impressed me with his dedication, integrity, and collaborative spirit. His expertise in Python, SQL, ElasticSearch, Data Engineering, and DevOps, coupled with his strong understanding of Scraping, OOP, API Development, and Git, make him a highly sought-after developer and an invaluable asset to any team.
Beyond his technical skills, Mehadi is an exemplary team player. His honesty, dedication, and willingness to collaborate make him a true asset to any team. He consistently goes the extra mile, shares his knowledge readily, and fosters a positive and productive work environment.
Without hesitation, I recommend Md. Mehadi Hasan for any backend or data-related system development project. He is a skilled developer, a dedicated team player, and a true asset to any organization.
