Menu

Docker

Going headless doesn't mean going down a blind alley! Use a dockerized IDE like Theia to code on you EC2 instance.

Mindful Coding in Headless Mode a.k.a. Kit Out Your EC2 Instance With a Dockerized IDE

, , , , , ,

Headless Is Not a Blind Alley Developing with an IDE on AWS EC2

I acquired an EC2 instance on Amazon’s cloud for building Generative AI models, gaining access to a Tesla T4 GPU and ample GPU RAM for deploying a Large Language Model (LLM) for an app developed for an investment banking firm. Access to the EC2 instance is primarily through the terminal on my MacBook, which starkly contrasts with the familiar Visual Studio Code (VS Code) environment that offers convenient integrations with GitHub and Docker.

While the Command Line Interface (CLI) is a valuable tool for developers, it can be cumbersome for tasks that benefit from a more visual interface. For quick changes to code moved from development to production environments, the EC2 instance running Ubuntu 22.04 OS offered only the Nano editor. Though Nano is a perfectly good editor, I wanted something closer to VS Code. The solution? A light version of the Theia IDE in Docker, which perfectly met my needs.

Ports, Posts and Portability Expose ports on EC2 instance

To access the EC2 instance, I SSH into it from an open terminal on my MacBook, using a downloaded .pem file for authentication. The command is as follows

	ssh -i ~/.ssh/secret.pem deusexmachina@45.245.72.29

A dockerized app typically listens for incoming requests at the host IP address on a specific port. When the app has a web UI, it can be accessed in a web browser using the IP address and port number, allowing access from any location on the web that can reach the IP address.

For our EC2 instance, we can find the required public IP address using AWS CLI. As for the port used by the app, we must ensure it is exposed to the public internet. We will use AWS CLI to check which ports are exposed to the public internet. If the required port is not exposed, we will use AWS CLI to expose it. Ensure you have AWS CLI installed on your computer; if not, follow the instructions here. 

	aws ec2 describe-security-groups --group-ids sg-08x6bdf6x665XcxeX | \
jq '.SecurityGroups[0].IpPermissions[] | select(.IpRanges[].CidrIp == "0.0.0.0/0")'

The command retrieves information about a specific security group in AWS EC2 and uses jq to filter the results. Let’s break down each part of the command:

AWS CLI Command

aws ec2 describe-security-groups --group-ids sg-08x6bdf6x665XcxeX

  • aws ec2 describe-security-groups: This is an AWS CLI command used to retrieve details about one or more security groups.
  • --group-ids sg-08x6bdf6x665XcxeX: This option specifies the security group ID for which details are being requested. In this case, the security group ID is sg-08x6bdf6x665XcxeX.

This command outputs information about the specified security group in JSON format.

Piping to jq

| jq '.SecurityGroups[0].IpPermissions[] | select(.IpRanges[].CidrIp == "0.0.0.0/0")'

  • | jq: This pipe sends the output of the aws ec2 describe-security-groups command to jq, which is a command-line JSON processor.
  • .SecurityGroups[0]: This selects the first (and likely only) security group from the list of security groups returned by the AWS CLI command.
  • .IpPermissions[]: This iterates over each IP permission rule within the selected security group. Each security group can have multiple IP permission rules, which specify which IP ranges are allowed or denied access.
  • select(.IpRanges[].CidrIp == "0.0.0.0/0"): This select function filters the IP permission rules to find those that have an IP range (CIDR block) of 0.0.0.0/0. The 0.0.0.0/0 CIDR block represents the entire IPv4 address space, indicating a rule that applies to all IP addresses.

Combining these parts, the full command performs the following actions:

  1. Retrieve Security Group Information: It fetches details about the security group with ID sg-08x6bdf6x665XcxeX using the AWS CLI.
  2. Process JSON Output with jq: It processes this JSON output to:
    – Access the first security group in the list.
    – Iterate over its IP permission rules.
    – Filter these rules to select only those that allow access from any IP address (0.0.0.0/0).

Here is the JSON indicative of a rule allowing TCP traffic on port 22 (typically used for SSH) from any IP address.

	{
  "IpProtocol": "tcp",
  "FromPort": 22,
  "ToPort": 22,
  "IpRanges": [
    {
      "CidrIp": "0.0.0.0/0"
    }
  ],
}

What if the port we need is not open? Here is the command to open a port.

	aws ec2 authorize-security-group-ingress --group-id sg-08x6bdf6x665XcxeX \
	--protocol tcp --port 8081 --cidr 0.0.0.0/0

The aws ec2 authorize-security-group-ingress command adds an ingress rule to a specified security group. An ingress rule specifies the type of incoming traffic that is allowed to reach instances associated with the security group. We have used the command with options and arguments as follows:

  • --group-id sg-08x6bdf6x665XcxeX: This option specifies the ID of the security group to which you want to add the ingress rule. In this case, the security group ID is sg-08x6bdf6x665XcxeX.
  • --protocol tcp: This option specifies the protocol for the rule. Here, tcp indicates that the rule applies to the TCP protocol.
  • --port 8081: This option specifies the port number to which the rule applies. In this case, it is port 8081.
  • --cidr 0.0.0.0/0: This option specifies the CIDR block (range of IP addresses) that is allowed by this rule. 0.0.0.0/0 represents all IP addresses, meaning the rule allows incoming traffic from any IP address.

Putting it all together, the command:

  1. Modifies the Security Group: It targets the security group with ID sg-08x6bdf6x665XcxeX.
  2. Specifies the Protocol: It applies to the TCP protocol.
  3. Defines the Port: It applies to port 8081.
  4. Sets the IP Range: It allows traffic from any IP address (0.0.0.0/0).

By running this command, we modify the specified security group to allow incoming TCP traffic on port 8081 from any IP address. In this way, we allow access to an application that is listening on port 8081 to users from anywhere on the internet.

Note on Security:

Allowing traffic from 0.0.0.0/0 means that the port is open to the entire internet. This can be a significant security risk. It is generally advisable to restrict the CIDR range to specific IP addresses or ranges that require access, rather than opening it up to all addresses.

Theia Use Case:

The Theia IDE runs as a web application that listens on port 3000. By running the command: aws ec2 describe-security-groups --group-ids sg-08x6bdf6x665XcxeX | jq '.SecurityGroups[0].IpPermissions[] | select(.IpRanges[].CidrIp == "0.0.0.0/0").FromPort', I found that port 3000 was among those open to TCP traffic. However, it was already in use by the web application Grafana. To resolve this, I decided to map port 3000 on Theia’s Docker container to port 3030 on the EC2 instance. I then ran the command to modify the security group, allowing HTTP (or any TCP) traffic to reach Theia on port 3030 from any IP address.

Let’s look at the installation process next.

Tango With Theia Installation with Docker

Here are the steps to install Theia:

  1. Pull the docker image: docker pull t1m0thyj/theia-alpine
  2. Spin up the docker container:
	docker run -d --init \
  --name theia_service \
  --restart always \
  -p 3030:3000 \
  -v /home/ubuntu:/home/project t1m0thyj/theia-alpine

The docker run command creates a container named theia_service and maps port 3030 on the EC2 instance to port 3000 on the container. The image t1m0thyj/theia-alpine is a lightweight version of the Theia IDE. With the container up and running, as verified by docker ps, we can access Theia using the URL http://45.245.72.29:3030 in a web browser. Here is what that looks like in Safari on my MacBook.

Use Docker to install Theia IDE on AWS EC2 instance. Since the container needs write permission to create or update text files on the shared volume, the directory permissions must be correctly set. Further, it may be necessary to create a user on the container with the same UID/GID as the user on the host. This can be conveniently done with options and arguments passed to the docker run command.
Use Docker to install Theia IDE on AWS EC2 instance. Since the container needs write permission to create or update text files on the shared volume, the directory permissions must be correctly set. Further, it may be necessary to create a user on the container with the same UID/GID as the user on the host. This can be conveniently done with options and arguments passed to the docker run command.

Using Docker to install Theia IDE on an AWS EC2 instance involves ensuring the container has the necessary permissions to create or update text files on the shared volume. The directory permissions must be correctly set, and it may be necessary to create a user on the container with the same UID/GID as the user on the host. This can be conveniently done with options and arguments passed to the docker run command.

Although the dockerized Theia running on EC2 was accessible this way, it did not allow creating or modifying files. In other words, it functioned as a read-only IDE!

Who Am I?  User Write Permissions Issue 

Using Docker to install Theia IDE on an AWS EC2 instance involves ensuring the container has the necessary permissions to create or update text files on the shared volume. The directory permissions must be correctly set, and it may be necessary to create a user on the container with the same UID/GID as the user on the host. Looking into the docker logs with the command docker logs theia_service on the EC2 machine, it became apparent that the container did not have sufficient permissions on the shared volume.

Ensure that the shared folder between the EC2 host and docker container has the correct ownership and permissions with the following commands:

	sudo chown -R ubuntu:ubuntu /home/ubuntu
sudo chmod -R 775 /home/ubuntu

Here, the logged-in user on the EC2 instance has name ubuntu. This can be verified by running the whoami command at the terminal prompt. Alternatively, use the $USER variable in the command like so: sudo chown -R $USER:$USER /home/ubuntu.

This alone wasn’t enough to fix the issue. Running the whoami command inside the Docker container from an interactive shell revealed the container user to be theia. The fix to the permissions error was to ensure that the container user theia has the same permissions as the instance user ubuntu. This is possible with a small modification to the options and arguments passed to the docker run command as follows:

	docker run --rm --init \
  --name theia_service_temp \
  -d -p 3030:3000 \
  -v /home/ubuntu/:/home/project \
  -u $(id -u ubuntu):$(id -g ubuntu) \
  t1m0thyj/theia-alpine

This command spins up a temporary container with --rm for testing the approach. The container is removed upon exit. 

The user on the docker container may not always align with the user on the host instance. This may prevent the application running in the container from creating or modifying files on the folder shared with the host. First, ensure that the folder on the host has the correct ownership and permissions. If this doesn't solve the problem, then further steps are needed to ensure that the user inside the container has the same permissions as the user on the host.
The user on the docker container may not always align with the user on the host instance. This may prevent the application running in the container from creating or modifying files on the folder shared with the host. First, ensure that the folder on the host has the correct ownership and permissions. If this doesn’t solve the problem, then further steps are needed to ensure that the user inside the container has the same permissions as the user on the host.

A tempfile was created using the IDE with content “Namaste ji!”. The changes were verified using cat \home\ubuntu\templfile on the host instance. Then the contents of the file were modified and changes verified. With this validation of the approach to fix the permissions error, the final docker run command was as follows:

	docker run --init \
  --name theia_service \
  --restart always \
  -d -p 3030:3000 \
  -v /home/ubuntu/:/home/project \
  -u $(id -u ubuntu):$(id -g ubuntu) \
  t1m0thyj/theia-alpine

Conclusions Key Takeaways

Going headless doesn’t mean having to forgo an IDE. With Docker, the installation process can be painless and fruitful.

Join our FastAI class to learn from experienced instructors who excel not only in building state-of-the-art models but also in tackling complex engineering challenges in real-world scenarios. Our instructors bring a wealth of expertise from successfully deploying applications that solve tangible problems and reach customers effectively. Through hands-on projects and personalized guidance, you’ll gain practical skills in AI and machine learning, ensuring you’re equipped to create impactful solutions in your career. Whether you’re aiming to advance your knowledge or pivot into a new field, our FastAI class offers the perfect blend of theoretical foundations and real-world application to help you succeed.

Grafana is an excellent way to quickly wrap interactive visualizations around a backend.

Get Grafana in Docker on Raspberry Pi and Share Compelling Visualizations

, , ,

Story-Telling With Grafana Create Dashboards With Interactive Visualizations 

Building Machine Learning or AI models is just the beginning. Conveying the insights derived from these models is equally, if not more, important. Interactive visualizations are powerful tools for gaining buy-in and influencing decisions. However, creating effective visualizations can often require as much effort as the modeling itself.

Get Grafana in Docker on Raspberry Pi and Share Compelling Visualizations

Use Grafana in Docker on Raspberry Pi

Grafana offers a low-code or no-code solution for creating dashboards. By connecting queries to your data backend, you can quickly develop visualization templates with live data feeds. Grafana enables you to generate compelling visualizations and craft a narrative that supports your recommended actions. For example, you can use gauges to track monthly spending against a budget and recommend cost-saving measures like eating in instead of dining out.

Installation Setting up Grafana in Docker 

ollow these steps to set up Grafana on your Raspberry Pi:

  1. Pull the Docker Image: docker pull grafana/grafana
  2. Create Folders for Persistence: grafana/data and grafana/provisioning; 
  3. Grant Access to Grafana User: chown 472:472 /path/to/grafana/data. 
  4. Run the Docker Container:
	docker run --name grafana_service \
	--restart always \
	-d -p 3000:3000 \
	-v /path/to/grafana/data:/var/lib/grafana \
	-v /path/to/grafana/provisioning:/etc/grafana/provisioning \
	-e GF_SECURITY_ADMIN_USER=admin \
	-e GF_SECURITY_ADMIN_PASSWORD=topsecretphrase \
	grafana/grafana

his command creates a Docker container named grafana_service with persistent data storage in shared volumes and sets up an admin account.

  1. Launch Grafana: http://localhost:3000

Sharing  Generate a Shareable Link

To share your Grafana dashboards:

  1. Generate a Shareable Link: Click on the blue button labeled “SHARE” to obtain a URL for sharing. Recent versions of Grafana also allow you to create a public URL.
  2. Update the URL for Intranet Sharing: The generated URL will contain ‘localhost’. Replace ‘localhost’ with the IP address of the host machine to share the dashboard on your local network. Use a free IP scanner like LanScan (for Mac) to find the IP address of machines on your intranet.

By following these steps, you can effectively share your Grafana dashboards, ensuring that your visualizations reach your intended audience.

Share the Grafana dashboard by generating a shareable link as shown. The localhost will need to be replaced with the host IP address. Use a tool like LanScan (Mac) to find the IP broadcast by machines on the local network.
Share the Grafana dashboard by generating a shareable link as shown. The localhost will need to be replaced with the host IP address. Use a tool like LanScan (Mac) to find the IP broadcast by machines on the local network.

With Grafana and Docker, setting up and sharing interactive visualizations on your Raspberry Pi has never been easier. Use these tools to create compelling dashboards that effectively communicate your data insights.

Join our FastAI class to master not only the art of building robust AI models but also the essential skills of visualizing results and deploying software seamlessly into the hands of customers. Our course, led by experienced Ph.D. instructors, covers the full spectrum of AI development—from crafting accurate models to creating interactive visualizations with tools like Grafana and managing deployments using Docker on platforms like Raspberry Pi. Gain the comprehensive expertise needed to turn data into actionable insights and deliver real-world applications efficiently.

Manage Docker containers using Portainer.

Manage Docker Containers on Your Raspberry Pi With Portainer

, , , , ,

Docker is addictive. As it becomes an indispensable tool in our toolkit, we need a way to streamline container management. Docker offers Docker Desktop, a GUI application, for Mac and Windows users. For Raspberry Pi, I recommend Portainer.

Docker offers Docker Desktop for Mac or Windows users. For Raspberry Pi running Raspbian, I recommend Portainer. Use it to start, stop, modify or remove containers and monitor usage statistics. Portainer runs in own container.
Docker offers Docker Desktop for Mac or Windows users. For Raspberry Pi running Raspbian, I recommend Portainer. Use it to start, stop, modify or remove containers and monitor usage statistics. Portainer runs in own container.

Portainer provides an intuitive web-based interface to manage Docker containers, making it easy to start, stop, modify, or remove containers and monitor usage statistics. It runs in its own container and is well-suited for single-board computers (SBCs) like the Raspberry Pi running Raspbian.

Here’s how to get Portainer up and running on your Raspberry Pi:

  1. Pull the image: docker pull portainer/linux-arm
  2. Spin up the container with docker run command like so:
	docker run --name portainer_service --network docker-net \
	-d -p 9000:9000 \
	-v /var/run/docker.sock:/var/run/docker.sock \
	-v ~/Your/path/to/data:/data \
	portainer/portainer:linux-arm
  1. Access Portainer: Open your web browser and go to http://localhost:9000. That’s it!

Now access Portainer from: http://localhost:9000. That is it!

Deploying AI Models in Real-World Situations

Being an AI practitioner means not only building models but also being familiar with various environments in which these models are deployed. Field applications often leverage platforms such as SBCs. The Raspberry Pi is a popular SBC that runs Linux and integrates seamlessly with various sensors and actuators.

In our FastAI course, you’ll learn from Ph.D. instructors who have invaluable expertise in both model building and the engineering required to deploy these models in real-world situations. We cover essential tools like Docker and Portainer, equipping you with the skills needed to put your AI applications into the hands of users effectively.

Join us to bridge the gap between developing AI models and deploying them in practical environments.

Dockerize data assets such as database and SQL client on Raspberry Pi.

Baking Data Apps Into Raspberry Pi

, , , , ,

In an earlier blogpost, we looked at why Docker is such an indispensable tool in the developer or data scientist’s toolkit, especially for shipping code that runs consistently across different environments. The Raspberry Pi, with its ARM architecture, provides a great example. Running software developed in traditional x86 environments on the Raspberry Pi can be challenging. Docker helps surmount these challenges and saves us endless frustration.

Show Me The Money Expense-Tracking App

I recently built an app for tracking expenses, particularly when my kid spent too much money on Zomato, a food delivery app! The idea was to set a monthly budget and track expenses against it. Visualizing data is crucial for self-adjustment on the path to improvement. The expense tracking system I built allows tracking expenses in common categories such as groceries, restaurants, utilities, etc.

The UI in PyQt6 for CRUD operations on Postgres warehouse. The MVC has Python with business objects in Pydantic and the SQLAlchemy ORM.
The expenses are managed through the app with a PyQt6 app that has a Postgres backend. The user can create a new expense, retrieve it for viewing, update it and apply changes or delete it. The Python implementation uses PyQt6 for UI and Pydantic for business objects with SQLAlchemy ORM.

I developed the app in the Python programming language. The app’s GUI, built in PyQt6, supports CRUD operations on a Postgres database. Users can create, view, update, and delete expenses. The architecture is a standard MVC framework with Pydantic business objects and SQLAlchemy ORM. 

Additionally, I built a dashboard in Grafana, a browser-based dashboarding tool. I developed the app on my MacBook and deployed it on the “always on” Raspberry Pi 400 at home.

Eat Humble Pie Code portability issues

I encountered many hurdles when porting code to the Raspberry Pi. The PyQt6 library couldn’t be installed in a virtual environment using the pip installer, so I installed it system-wide from the Debian repository. This prevented running my Python code in the virtual environment. Consequently, other dependencies like Pydantic and SQLAlchemy also had to be installed system-wide.

I built an expense tracking application to monitor day-day expenses and generate informative summaries. I used Grafana to construct a dashboard featuring interactive visualizations.
Get Grafana on Raspberry Pi. Build and share interactive visualizations.

I pulled the official Docker image for Postgres to get the database running. See this blogpost. However, finding a suitable SQL client was surprisingly difficult. I tried TablePlus, DBeaver, and Pgdmin, but each had issues post-installation, rendering them unusable.

A bad workman always blames his tools. Was I using the wrong tools or is the Raspberry Pi just Rubbish Pi?

It's Raining Containers Containerize the asset

The Raspberry Pi, being resource-constrained compared to a desktop, poses challenges in installing and running software. Docker can be invaluable in such situations. I was already running Postgres in a Docker container on the Raspberry Pi and found ARM-compatible containers for Pgadmin and Grafana. The pgAdmin container, provided by Elestio, is available as elestio/pgadmin.

Here’s how to install it:

  1. Pull the image: docker pull elestio/pgadmin
  2. Run the container:
	docker run --name pgadmin_service \
	-d -p 8080:8080 \
	--network docker-net \
	-e PGADMIN_DEFAULT_EMAIL=handsomest.coder@email.com \
	-e PGADMIN_DEFAULT_PASSWORD=topsecret \
	-e PGADMIN_LISTEN_PORT=8080 \
	-v ~/path/to/pgadmin/servers.json:/pgadmin4/servers.json \
	elestio/pgadmin

This command starts a Pgadmin container named pgadmin_service in the background, accessible via http://localhost:8080, connected to the docker-net network, using the specified email and password for initial login credentials, and mounting a servers.json file for predefined server connections.

Don't Dally, Do Docker  Summary

Using Docker images for Pgadmin and Postgres provided a smooth path to setting up these vital data assets on my Raspberry Pi. This approach should generally be the default for installing software in app development. For example, when building home automation systems, it’s beneficial to containerize services like Mosquito MQTT, Node-Red, InfluxDB, and Grafana that are part of developer’s IoT stack. Refer to this YouTube video for a guide. For managing all these containers with a graphical UI, use Portainer, available as a free community edition from Docker. 

Unlock the full potential of AI with our FastAI course, where you’ll not only master AI integration into applications but also gain essential skills like Docker, crucial for deploying and managing your apps in real-world environments. Learn to bridge the gap between development and practical implementation, ensuring your AI solutions are robust, scalable, and ready for end-users. Join us to transform your technical expertise into powerful, customer-ready applications.

Docker containers make code easy to ship

Docker. Why bother?

, , , , , , , , , , ,

Back in the day, when I wrote apps in C/C++, I compiled the code into an executable for shipping. When we code in Python, how do we ship code?

We could simply send our code to the customer to run it on their machine. But the environment in which our code would run at the customer’s end would almost never be identical to ours. Small differences in environment could mean our code doesn’t run and debugging such issues is a colossal waste of time, not to mention repeating the process for every customer.

But there is a better way and that is Docker!

Microservice Web-Crawling Spider

The micro service, which is part of a larger app, features a spider that crawls web domains for content that it warehouses in a postgres database.
The micro service, which is part of a larger app, features a spider that crawls web domains for content that it warehouses in a postgres database.

Here is a microservice I built that was part of a larger app. It has a spider that crawls multiple web domains for content that it scrapes and puts into a Postgres warehouse. I built out the spider in Scrapy framework in Python 3 and used the psycopg2 client for database CRUD operations.

Shipping the code means replicating the environment on the machine where it will run. In the process, small changes may creep up. The version of Python or its dependencies may differ. The version of Postgres may also differ. The devil lies in details! Small differences can throw a spanner in the works. That is why, shipping code in this manner is not recommended.

Instead, dockerize the app!

Ship It With Docker Encapsulate Code for Cross-Platform Deployment

Let’s start by dockerizing the Postgres warehouse. The steps are pulling the docker image from docker hub and then spinning up the container!

Pull the image like so: docker pull postgres

Then spin up the container like so: docker run --name postgres_service --network scrappy-net -e POSTGRES_PASSWORD=topsecretpassword -d -p 5432:5432 -v /Your/path/to/volume:/var/lib/postgresql/data postgres

This command not only launches the container but also connects it to the Docker network for seamless communication among containers. (Refer this blogpost.) Additionally, it ensures data persistence by sharing a folder between the host machine and the container.

Let’s break down the docker run command into its constituent parts:

  1. docker run: This is the command used to create and start a new container based on a specified image.
  2. --name postgres_service: This flag specifies the name of the container. In this case, the container will be named “postgres_service”.
  3. --network scrappy-net: This flag specifies the network that the container should connect to. In this case, the container will connect to the network named “scrappy-net”.
  4. -e POSTGRES_PASSWORD=topsecretpassword: This flag sets an environment variable within the container. Specifically, it sets the environment variable POSTGRES_PASSWORD to the value topsecretpassword. This is typically used to configure the containerized application.
  5. -d: This flag tells Docker to run the container in detached mode, meaning it will run in the background and won’t occupy the current terminal session.
  6. -p 5432:5432: This flag specifies port mapping, binding port 5432 on the host machine to port 5432 in the container. Port 5432 is the default port used by PostgreSQL, so this allows communication between the host and the PostgreSQL service running inside the container.
  7. -v /Your/path/to/volume:/var/lib/postgresql/data: This flag specifies volume mapping, creating a persistent storage volume for the PostgreSQL data. The format is -v <host-path>:<container-path>. In this case, it maps a directory on the host machine (specified by /Your/path/to/volume) to the directory inside the container where PostgreSQL stores its data (/var/lib/postgresql/data). This ensures that the data persists even if the container is stopped or removed.
  8. postgres: Finally, postgres specifies the Docker image to be used for creating the container. In this case, it indicates that the container will be based on the official PostgreSQL image from Docker Hub.

Code Contained Make Container From Code With Dockerfile

For creating a container from own code – Python scripts and dependencies, there are a few steps. The first step is creating a dockerfile. The dockerfile for our Scrapy app looks like so:

	# Use the official Python 3.9 image
FROM python:3.9

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install required dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Set the entry point command for running the Scrapy spider
ENTRYPOINT ["scrapy", "crawl", "spidermoney"]

This Dockerfile automates the process of building the image and running the container. It ensures that the container has all the necessary dependencies to execute the Python app.

Let’s break down each line of the Dockerfile:

  1. FROM python:3.9: This instruction specifies the base image to build upon. It tells Docker to pull the Python 3.9 image from the Docker Hub registry. This image will serve as the foundation for our custom image.
  2. WORKDIR /app: This instruction sets the working directory inside the container to /app. This is where subsequent commands will be executed, and it ensures that any files or commands are relative to this directory.
  3. COPY . /app: This instruction copies the contents of the current directory on the host machine (the directory where the Dockerfile is located) into the /app directory within the container. It is a common practice to place the dockerfile in the project directory at the top level, for including application code and files inside the Docker image.
  4. RUN pip install --no-cache-dir -r requirements.txt: This instruction runs the pip install command inside the container to install the Python dependencies listed in the requirements.txt file. The --no-cache-dir flag ensures that pip doesn’t use any cache when installing packages, which can help keep the Docker image smaller.
  5. ENTRYPOINT ["scrapy", "crawl", "spidermoney"]: This instruction sets the default command to be executed when the container starts. It specifies that the scrapy crawl spidermoney command should be run. This command tells Scrapy, a web crawling framework, to execute a spider named “spidermoney”. When the container is launched, it will automatically start executing this command, running the Scrapy spider.

The dockerfile is a recipe. The steps to prepare the dish are as follows:

  1. Build the image with docker build -t scrapy-app .. The dockerfile is a series of instructions to build a docker image from. The build process downloads the base layer and adds a layer with every instruction. Thus, layer by layer, a new image in constructed which has everything needed to spin up a container that runs the app.
  2.  Spin up the app with the docker run command. For example: docker run --name scrapy_service --network scrappy-net -e DB_HOST=postgres_service scrapy-app. This command creates container named ‘scrapy_service’ from the image ‘scrapy-app’ and connects it to the network ‘scrappy-net’. The name of the container running Postgres is passed as an environment variable with -e flag to configure the app to work with the database instance.

Pushmi-Pullyu Push to and Pull From Hub

Launch the microservice by starting the docker containers. First, start 'postgres_service'. Then start 'scrapy_service'. The latter will scrape content, populate the database and exit.
Launch the microservice from the docker dashboard. Start the Postgres container (postgres_service) first and then the app container (scrapy_service). The app will run and exit when it has completed the scraping task. Check the database for new records added.

Once the microservice is containerized, launching it is as simple as starting the containers. Start the Postgres container first, followed by the app container. This can be easily done from the Docker dashboard.

Verifying Deployment:

Verify the results by examining the Postgres database before and after running the microservice. Running SQL queries can confirm that the spider has successfully crawled web domains and added new records to the database.

Postgres database Tableview client SQL query snapshot after microservice has run
SQL query shows the before-after state of the Postgres database. The Python Scrapy spider has crawled web domains and added new records.
Postgres database Tableview client SQL query snapshot before microservice has run
SQL query shows the before-after state of the Postgres database. The Python Scrapy spider has crawled web domains and added new records.

The figures show that 1232 records were added in this instance.

Now shipping the code is as simple as docker push to post the images to docker hub followed by docker pull on the target machine.

Conclusion The Docker Revolution

With Docker, shipping code becomes a streamlined process. Docker encapsulates applications and their dependencies, ensuring consistency across different environments. By containerizing both the database and the Python app, we simplify deployment and guarantee reproducibility, ultimately saving time and effort.

In conclusion, Docker revolutionizes the way we ship and deploy code, making it an indispensable tool for modern software development.

Being in the AI profession is more than just coding neural networks. Getting them into the hands of customers demands a thorough understanding of contemporary microservices architecture patterns. Learn from experienced instructors who can be your guide through our comprehensive coaching program powered by FastAI. Gain insights into cutting-edge techniques and best practices for building AI applications that not only meet the demands of today’s market but also seamlessly integrate into existing systems. From understanding advanced algorithms to mastering deployment strategies, our Ph.D. instructors will equip you with the skills and knowledge needed to succeed in the dynamic world of AI deployment. Join us and take your AI career to new heights with hands-on training and personalized guidance.

Docker network provides accessibility of one docker container to another; Containers networked in this way can exchange messages with each other.

Access One Docker Container From Another in Microservices Architecture

, , ,

Suppose I have two docker containers on a host machine- an app running in one container, requiring the use of database running in another container. The architecture is as shown in the figure.

Access One Docker Container From Another in Microservices Architecture

In the figure, the Postgres container is named ‘postgres_service’ and is based on official postgres image on docker. The data reside on a shared volume with local host. In this way, data are persisted even after container is removed.

The app container is named ‘scrapy_service’ and is based on an image created starting from official Python 3.9 base image for linux. The application code implements a web-crawler that scrapes financial news websites.

The web-scraper puts data into the postgres database. How to access postgres?

On the host machine, the postgres service is accessible at ‘localhost’ on port 5432. However, this will not work from inside the app container where ‘localhost’ is self-referential.

Solution? We create a docker network and connect both containers to it.

Create docker network and connect postgres container to it. Inspect and verify.
Spin up app’s container with connection to the network.

Create Docker Network Connect Postgres container to the network

  1. docker create network scrappy-net creates network named scrappy-net.
  2. docker network connect scrappy-net postgres_service connects the (running) postgres container to the network.
  3. docker network inspect scrappy-net shows the network and what’s on it. 

We now have a network ready to accept connections and exchange messages with other containers. Docker will do the DNS lookup with container name.

Spinup App Container  Link app to Postgres container

  1. docker build -t scrapy-app . builds the image named scrapy-app. The project directory must have dockerfile and requirements manifest. The entry-point that launches spider is scrapy crawl spidermoney or scrapy crawl spidermint.
  2. docker run –-name scrapy_service –-network scrappy-net -e DB_HOST=postgres_service scrapy-app spins up the container with connection to the network. It launches the crawler as per the entry-point spec. The container exits when the job is done. Thereafter, it can  be re-run as docker start scrapy_service with persistent network connection.
  3. docker logs scrapy_service > /Users/sanjaybhatikar/Documents/tempt.txt 2>&1 saves the app’s strreamed output on stderr and stdout to temp text file for inspection.