Who Took the Cookie From the Cookie-Jar? Session Management in Scraping
Ever wondered how to efficiently scrape massive websites with dynamic content? Scrapy is a simple yet powerful tool that has stood the test of time. By sending an initial request to the landing page, the spider follows a chain of responses, each triggering new requests and scraping data as it goes. Scrapy’s asynchronous nature means that multiple requests can be processed concurrently, speeding up the entire process. Combined with automatic session management, it ensures seamless navigation through complex websites. The scraped data is then processed and stored for future analysis, making Scrapy an indispensable tool for any production-scale web scraping project.
Tee and Toast With My Post
How do databases handle massive data seamlessly, and how can you log commands without losing visibility?
When it comes to databases, PostgreSQL’s TOAST (The Oversized-Attribute Storage Technique) ensures robust performance by compressing and storing large data values—like lengthy text or binary files—outside the main table. This guarantees efficient storage and fast operations, letting PostgreSQL handle big data with ease and reliability.
On the Linux side, the `tee` command offers unmatched flexibility for managing output. It allows you to redirect command output to both the terminal and files simultaneously, making it perfect for logging or debugging without losing visibility. Trusted and widely used, TOAST and `tee` provide reliable solutions to modern data and system challenges.
So how about some tee with your TOAST?
Unlocking the Power of Language: Leveraging Large Language Models for Next-Gen Semantic Search and Real-World Applications
Invited talk at Calfus, Pune, June 20, 2024.
Mindful Coding in Headless Mode a.k.a. Kit Out Your EC2 Instance With a Dockerized IDE
In “Mindful Coding in Headless Mode,” we explore how to enhance your coding experience on an AWS EC2 instance by setting up a Dockerized Theia IDE. Developing AI models on EC2 provides powerful resources but often lacks the familiar, integrated development environment (IDE) we rely on for efficiency. This blog post details my journey from the limitations of CLI-based editing to achieving a seamless development experience akin to using VS Code. By leveraging Docker and Theia, I transformed my EC2 instance into a powerful, user-friendly development platform that allowed me to efficiently deploy and manage a Large Language Model (LLM) for an investment banking app. Whether you’re managing quick code changes or deploying complex applications, this guide will show you how to bring the full functionality of a modern IDE to your headless EC2 instance, ensuring you’re not just coding, but coding smartly and effectively.
An Interesting SQL Query or Two
Mastering SQL is essential for anyone building data-driven models. In the Age of AI, SQL might seem outdated, but it’s actually crucial for harnessing the power of data to create compelling narratives. In this blog post, we explore advanced SQL queries through the lens of a real-life use case, demonstrating its ongoing relevance and indispensability.
Get Grafana in Docker on Raspberry Pi and Share Compelling Visualizations
Grafana offers a powerful, low-code solution for creating interactive dashboards that convey insights effectively. By deploying Grafana in Docker on a Raspberry Pi, you can quickly set up and manage compelling visualizations using live data feeds. This guide provides step-by-step instructions for installing Grafana, configuring persistent storage, and setting up an admin account. Additionally, it explains how to generate and share dashboard links within your local network, making it easy to communicate your data-driven recommendations. Whether you’re tracking monthly spending or showcasing AI model results, Grafana simplifies the process of turning data into actionable insights.
Manage Docker Containers on Your Raspberry Pi With Portainer
Docker is an essential tool for developers, and managing containers on a Raspberry Pi can be streamlined with Portainer. While Docker Desktop provides a GUI for Mac and Windows, Portainer offers an intuitive web-based interface for Raspberry Pi running Raspbian. It allows users to start, stop, modify, or remove containers and monitor usage statistics efficiently. Running in its own container, Portainer is well-suited for single-board computers like the Raspberry Pi. Setting it up involves pulling the Portainer image and running a simple Docker command, after which it can be accessed via http://localhost:9000.
Baking Data Apps Into Raspberry Pi
**Excerpt: Baking Data Apps Into Raspberry Pi**
Docker is an indispensable tool for developers, ensuring code runs consistently across different environments. This is especially true for the Raspberry Pi, with its ARM architecture, where Docker can help overcome challenges of running software developed in x86 environments. In this blog post, I discuss building an expense-tracking app using Python, PyQt6, and Postgres, and how Docker was essential in deploying it on a Raspberry Pi. I also cover setting up a Grafana dashboard for data visualization and the hurdles encountered with code portability. Finally, I highlight the advantages of containerizing applications on the Raspberry Pi to streamline installation and management, using tools like Docker and Portainer.
Docker. Why bother?
Shipping Python applications can be challenging due to varying environments on different machines, but Docker offers an elegant solution by encapsulating code and dependencies into containers. This blogpost details the development of a microservice using Scrapy to crawl web domains and store data in a Postgres database, showcasing the ease of deployment with Docker. By dockerizing both the database and the Python app, deployment becomes seamless, ensuring consistency and saving time. Learn how to use Dockerfiles for building images, running containers, and managing microservices, making Docker an indispensable tool for modern software development.
Why RAG Is All the Rage in Generative AI
Continuous monitoring of the business landscape is imperative for any company, ensuring the identification of both threats and opportunities. This entails prospecting opportunities and gathering competitor intelligence, synthesized into actionable executive briefs by analysts. The task involves navigating through diverse information sources like websites, regulatory filings, social media, and news articles, contributed by various stakeholders. Recent advancements in Generative AI and Large Language Models (LLMs) offer automation possibilities for insight generation. We developed a chatbot assistant for Bayer Crop Science USA, leveraging the “co-pilot” mode of assistive AI to enhance productivity of the prospecting and competitor intelligence team and mitigate the risk of missed opportunities.
Accelerate Key Parts of Biotech R&D Pipeline Using Computer Vision
In the realm of Biotech R&D, the cultivation of genetically engineered plants through tissue culture is a pivotal process, but low success rates have long hindered productivity. Enter a revolutionary solution: a Convolutional Neural Network (CNN) designed to identify non-transformable embryos within the initial two weeks, preventing the inefficient use of resources and bottlenecking in the downstream pipeline. By leveraging deep learning models and an extensive dataset, this computer vision approach not only enhances throughput but also accelerates time-to-market, ushering in a new era of efficiency and innovation in biotech research and development.
Access One Docker Container From Another in Microservices Architecture
**Excerpt: Access One Docker Container From Another in Microservices Architecture**
In a microservices architecture, connecting an app container to a database container on the same host can be challenging. This blogpost demonstrates how to achieve this by creating a Docker network to facilitate communication between containers. Using a Postgres container named ‘postgres_service’ and an app container ‘scrapy_service’, the article guides you through setting up the network, connecting the containers, and configuring the app to communicate with the database. This setup ensures seamless data exchange and persistence, streamlining the deployment of interconnected microservices.
Address Privacy Concerns in Usage of Generative AI With Data Privacy Vault
The rise of generative AI, including large language models (LLMs), has sparked widespread discussions. While there’s immense excitement about the transformative potential of generative AI in various fields, such as product development and content creation, there are substantial privacy and compliance challenges. Unlike traditional applications, LLMs lack a “delete” button, posing risks for businesses facing data privacy regulations. A solution proposed is the implementation of a data privacy vault, a concept gaining popularity in traditional data management. This vault isolates and protects sensitive customer data, addressing challenges associated with LLMs’ inability to selectively delete or “unlearn” specific data points. The article delves into the intricacies of generative AI’s impact on privacy, compliance hurdles, and suggests a novel approach grounded in the IEEE’s recommended architecture for managing sensitive customer data. This approach involves using a data privacy vault to prevent the leakage of sensitive data into LLMs, ensuring compliance and controlled access to information. By incorporating this vault into the model training and inference pipelines, businesses can leverage the power of AI without compromising on privacy.
Unorthodox Onomastics: How AI Can Assist You in Discovering the Perfect Name for Your Child
In exploring the collection of names generated by a Deep Learning model, we’ve unveiled the versatility of Large Language Models (LLMs) with their massive parameters and intricate architectures. Beginning with the foundational bigram model, our journey transitions into the realm of deep neural networks, where the introduction of embeddings allows for a nuanced understanding of language. Beyond the task of generating names, this evolution highlights the potential of language models to grasp the intricacies of human communication. As we take these initial steps, we glimpse into a future where machines, armed with sophisticated language comprehension, play a pivotal role in diverse applications, paving the way for an era of enhanced human-machine interaction.
A Mondrianesque Masterpiece with App Layouts in AppInventor
AppInventor’s simplicity can yield striking complexity; much like the art of Piet Mondrian, it allows for the creation of aesthetically pleasing, grid-like layouts. Despite the simplicity of the humble rectangular and vertical arrangements, you can craft an app’s visual landscape with the precision and elegance reminiscent of the iconic artist’s work.
The Essence of Event-Driven Programming in Android App Development
Event-driven programming in the context of Android app development is akin to the anticipation of a package delivery. Just as you can either vigilantly watch your doorstep or await the doorbell’s chime to signal your package’s arrival, programming offers two paradigms. The event-driven paradigm, symbolized by the doorbell scenario, is constructed as a “mad-lib” of “WHEN event occurs, DO action,” where the first part represents the event, and the second part is the callback or action triggered in response. Android apps rely on this paradigm, executing callback functions when specific events transpire, whether driven by user interactions like tapping or swiping, or the passage of time through clocks and timers. Consider the “Mole Mash” app, where a green event-handler block initiates a timer, and a purple callback block moves the mole to random coordinates, breathing life into the game. This dynamic approach empowers developers to craft responsive applications accommodating diverse user experiences.
Building a Deep Learning Model From Scratch in PyTorch
We continue our quest to delve deeper into the realm of deep learning, refactoring the core functions of our simple neural network to engineer a deep learning model. By introducing the flexibility to customize the number of hidden layers and neurons in each layer, we elevate our model’s capacity for learning intricate patterns in the data. Our code refactoring maintains the simplicity of the six fundamental functions, preserving a user-friendly interface while embracing the power of deep learning. The journey doesn’t end here, as we’re poised to explore the innovations that have made deep learning models more practical and dependable, from techniques like batch normalization and residual network connections to the valuable concept of dropout. As the field of deep learning continues to advance, we eagerly anticipate diving even deeper into its depths.
Building a Simple Neural Network From Scratch in PyTorch
In this post, we explore the process of building a simple neural network from scratch in PyTorch for binary classification. We discuss the network’s architecture, including the use of ReLU activation in the hidden layer and sigmoid activation in the output layer. We emphasize the importance of data partitioning for training, validation, and testing. The neural network is trained using gradient descent, and we provide insights into initializing weights and biases, implementing forward and backward passes, and updating coefficients. Furthermore, we explain the significance of non-linear activations in making the network a universal function approximator. We demonstrate an approach that is easily extensible to deep learning simply by refactoring code while keeping the same set of functions and interfaces implemented here. Thus, we offer a versatile and scalable neural network architecture.
Unleash Your Creativity with MIT AppInventor: Build Awesome Apps That Amaze Friends
Discover the incredible world of MIT AppInventor, where app development becomes a breeze, even for those without coding experience. Created by MIT and Google, this platform has empowered over 18 million users to craft a staggering 85 million apps, proving that anyone can bring their ideas to life with the right tool. From interactive show-and-tell experiences to animated games and AI-powered marvels, AppInventor simplifies app creation through its user-friendly drag-and-drop interface and event-driven architecture. Dive into this vibrant community, harness your creativity, and unlock your app potential with MIT AppInventor.
Fire Up Your App With Firebase Firestore in MVVM Architecture With Jetpack Compose (Part V)
The repository opens a door to data storage in the Data layer of our Android app in MVVM architecture. The Domain layer houses our domain assets such as data models and the business logic. Adhering to the principle of programming to an interface, the Domain layer is where we define an interface containing the abstract functionality we need for data management, using only domain assets. The implementation requires we use the data models and CRUD functions in the Data layer. Adhering to the principle that the Domain layer does not know about other layers but other layers know about Domain layer. we flesh out the implementation of the repository interface in the Data layer’s package “repository”. By using the repository pattern, we put all the overhead of data management – error handling, asynchronous I/O, etc., in one place.
Fire Up Your App With Firebase Firestore in MVVM Architecture With Jetpack Compose (Part IV)
The different layers of MVVM architecture each require that we structure data differently, so we end up with multiple sets of data models representing the same information. In this article, we examine the drivers behind the need for distinct sets of data models. We introduce the “mapper” package within the Data layer which houses the code for mapping between these different sets of data models. We demonstrate a pattern for writing a mapper as an extension function of the source model, illustrating with an example. We conclude by setting the stage for the repository pattern which opens the door to the data in the app’s storage.
Fire Up Your App With Firebase Firestore in MVVM Architecture With Jetpack Compose (Part III)
The Domain layer serves as a vital bridge between the Data and Presentation layers. This intermediary layer decouples the logic of other layers from each other, promoting separation of concerns while facilitating a seamless data flow. The need for this bridge arises from the distinct design considerations when it comes to storing and working with data in the app. We explore the data models in Domain layer, mirroring their counterparts in the Data layer’s “remote” package. These models serve as the foundation for shaping data for processing and presentation in our app. We delve into the model attributes that enable delivery of a modular quiz with multimedia content. We take this opportunity to examine the trade-offs in choice of FIrebase storage and present an alternative.
Fire Up Your App With Firebase Firestore in MVVM Architecture With Jetpack Compose (Part II)
We delve into Firestore CRUD operations, where CRUD stands for Create, Retrieve, Update, and Delete, essential for data management. We explore how to integrate Firestore storage into our Android quiz app, following the Model-View-View Model (MVVM) architecture. Our journey includes defining an interface for data operations, creating data models, and implementing Firestore queries, while highlighting the crucial role of the Domain layer in shaping data into information for consumption.
Fire Up Your App With Firebase Firestore in MVVM Architecture With Jetpack Compose (Part I)
In this series, we explore the development of a scalable Android quiz app, embracing the Model-View-ViewModel (MVVM) architecture. Our app, “Android Ally,” utilizes Jetpack Compose for the user interface and Firebase Firestore for data storage, with Dagger-Hilt for Dependency Injection. At this stage, we assume the Android project is set up with Firebase dependencies, and Firestore contains sample data for modules and quizzes. The app’s primary focus is to facilitate native Android development learning through quizzes organized by modules, each containing questions, answers, and ranks. The MVVM architecture empowers the app with three layers: Data for storage, Presentation for the user interface, and Domain as the bridge between them. We examine code organization within these layers to comprehend the app’s structure, starting with Firebase Firestore client-side code for data retrieval.
A Guide to Enhancing User Experience by Integrating Image Picker Functionality in Your Android App
Alice, our intrepid explorer of Android Wonderland, had a grand vision for her Vaxin app. She dreamed of allowing users to personalize their experience by choosing an image from their phone’s gallery. Little did she know the adventures that awaited her. Part 1: Alice’s Journey in the Android Wonderland With unwavering determination, Alice declared a […]
Elevate Your App’s Aesthetics: Crafting a Vibrant and Tasteful Color Scheme with Material Theme
Material 3 is Google’s latest design system that brings a refreshed look and feel to Android applications. One of its key components is theming, which includes defining a cohesive color scheme for your app. This guide will focus on using Material 3 for theming, with a particular emphasis on colors, composition, and light and dark […]
Unleashing the Power of CNNs on Non-Image Data: A Creative Twist
We can be incredibly creative with CNNs. They might be designed for images, but with a little ingenuity, we can make them our allies in understanding non-image data like heart sounds. We don’t need to reinvent the wheel and build entirely new models. Instead, we can convert our data into an image format and let CNNs do what they do best. As we keep exploring the possibilities of deep learning, this approach opens exciting doors for improving medical diagnosis and treatment. So, remember, when dealing with non-image data and CNNs, sometimes all you need is a little creative twist.
Learning Like a Little Leaguer: Applying David Perkins’ Seven Principles of Teaching in Craft With Code
David Perkins’ seven principles of teaching, rooted in his childhood baseball experiences, provide valuable insights for hands-on learning environments like Craft With Code. By encouraging students to “play the whole game” by creating projects from the start, making learning relevant and engaging, and emphasizing targeted practice, educators can enhance the effectiveness of hands-on courses. These principles remind us that, much like little leaguers learning baseball, students in Craft With Code courses benefit from grasping the broader context before fine-tuning their skills – a philosophy that transforms hands-on learning into a transformative educational experience.
Vaxin: App for Keeping a Child’s Vaccinations on Track
Ensuring your child receives timely vaccinations is crucial for their health and well-being. However, managing the schedule for 24 vaccinations within the first 18 months of your child’s life can be a daunting task for parents. That’s where the Vaxin app comes to the rescue. With Vaxin, you can effortlessly keep track of your child’s vaccination schedule, leaving you free from the stress of keeping everything in mind. Simply enter your child’s birthday, and you’re all set to ensure they receive the vaccinations they need on time, promoting their health and peace of mind for parents.
Handling UI Events in Jetpack Compose
In the world of Android app development, effective event-handling is crucial for providing users with a seamless and engaging experience. To tackle the diverse range of user actions, from button clicks to text input and more, the Model-View-ViewModel (MVVM) architecture offers an elegant solution. The `onEvent` function within the View Model acts as a central hub for managing these events. However, the challenge lies in passing these diverse events to `onEvent` while preserving their unique flavors. This challenge is met with the introduction of a sealed class which serves as a versatile wrapper for various events, providing a common interface. This class not only simplifies event management but also ensures that the distinctive characteristics of each user action are retained. Through use of the mechanism described here, Android app developers can streamline event-handling and create exceptional applications that respond seamlessly to user interactions.
Unlocking App Inventor’s Event-Driven Magic for Crafting Interactive Mobile Apps
In the world of programming, the event-driven paradigm, much like patiently waiting for a doorbell to ring when expecting a package, is a cornerstone of creating dynamic and responsive mobile applications. It involves crafting code that responds to specific events or triggers, bringing interactivity to your apps. Through our exploration and mastery of this event-driven magic using App Inventor’s powerful tools and techniques, you’ll discover how to design and build engaging mobile apps that captivate users, transforming your ideas into interactive digital experiences.
Remote Work Monitoring: Ethics in the cat-and-mouse game of surveillance and tech-savvy workers”
In the evolving landscape of remote work, the issue of monitoring employee productivity while respecting ethical boundaries has taken center stage. The cat-and-mouse game between surveillance measures and tech-savvy workers adept at finding creative solutions has illuminated the need for a delicate balance. This article delves into the intricacies of remote work monitoring, shedding light on the ethical considerations that arise as managers and employees navigate this complex terrain, where trust and transparency are pivotal in maintaining a harmonious and productive remote work environment.
Groove with Gradio or Jive with FastAPI
In this rhythmic ode to coding tools, we explore the prowess of Gradio, FastAPI, and Streamlit in the world of technology. Gradio is celebrated for its speed and simplicity, making it an inspiring muse for developers. FastAPI’s power to shape endpoints and its flexibility are likened to a sage’s wisdom, constantly amazing with its versatility. Streamlit’s charm lies in its compromise, making it an ideal companion for day and night app development. The message encourages the reader, referred to as “tech bro,” to embrace these tools, and not be shy but boldly utilize them to architect their tech journey.
“Now that I have built my amazing AI app, how do I put it in the hands of my users?”
Two words, Rest API, unveil the digital alchemy that forges connections in the vast tapestry of the World Wide Web. “Rest,” a guiding acronym for Representational State Transfer, offers a blueprint for harmonious data exchange among machines, encapsulating the ever-changing states of systems like the ebb and flow of stock market data. “API,” or Application Programming Interface, serves as the digital handshake uniting denizens of the internet, be they human or machine. In a virtual universe rife with chaos, protocols emerge as the steadfast guardians of order, and the HTTP protocol acts as the conductor orchestrating seamless data exchanges. Yet, the Rest API emerges as a beacon of efficiency, offering answers to questions posed through a simple URL, unlocking the magic of automation. As the digital sentinel facilitating data’s journey, Rest API wields the power to transform ordinary applications into extraordinary AI marvels, and the FastAI coaching program stands ready to equip Python enthusiasts with the tools to navigate this exhilarating realm.
How many lines of code does it take to train a deep learning model?
Imagine this: With just a few lines of code, you can awaken a deep learning marvel, a digital creature that can recognize pets in the blink of an eye. The beauty of it all lies in the fact that you don’t need to be a mathematical wizard, drown in a data deluge, or mortgage your future for a supercomputer. It’s a digital adventure that levels the playing field, opening doors to anyone with the curiosity and a modest amount of programming experience.
Joy of Making
At the age of three, I discovered my love for creation through LEGO, which later evolved into a passion for technology and coding. This journey inspired the founding of Craft With Code, a platform dedicated to experiential learning. In an era of rapid technological advancement, our approach empowers students to actively create, fostering problem-solving skills and a digital mindset. We bring Confucius’ wisdom to life: “I hear and I forget. I see and I remember. I do and I understand.” Our students don’t just write code; they make it come alive on devices, equipping them with a versatile toolbox of modern programming skills.