Browsed by
Author: Afzal Badshah, PhD

Real-Time Data Processing: Taming the Flood of Data from a Connected World

Real-Time Data Processing: Taming the Flood of Data from a Connected World

Comments 30 Comments

Traditional data processing methods, which involve collecting and analyzing data in batches at specific intervals, are simply struggling to keep pace with this ever-increasing data flow. This is where real-time data processing comes into play. It’s a method of processing data streams at near-instant rates, enabling organizations to gain insights and make decisions based on the latest information as it becomes available. Visit the detailed tutorial here. Real-Time Data Processing Real-time data processing is a method of analyzing and interpreting…

Read More Read More

Matrix Multiplication on Multi-Processors: MPI4PY

Matrix Multiplication on Multi-Processors: MPI4PY

Comments 37 Comments

In this scenario, each processor handles a portion of the matrices, performing computations independently, and then the results are combined to obtain the final result. This parallelization technique leverages the capabilities of multiple processors to expedite the overall computation time.  Code: Explanation Import MPI Module and Initialize MPI Environment This line imports the MPI module from the mpi4py package, enabling the use of MPI functionalities. These lines initialize the MPI environment. MPI.COMM_WORLD creates a communicator object representing all processes in…

Read More Read More

Data Modeling and Schema Design in MongoDB

Data Modeling and Schema Design in MongoDB

Comments 27 Comments

Data modelling and schema design are pivotal aspects of MongoDB database management, crucial for structuring data effectively to meet application requirements. In this tutorial, we’ll explore the fundamentals of data modelling and schema design in MongoDB through practical examples set in a Pakistani context. Visit the detailed tutorial here. Data Model Design Modeling in NoSQL refers to the process of designing how data will be structured and organized within a NoSQL database. Unlike traditional relational databases, NoSQL databases offer more…

Read More Read More

Parallel Summation using MPI in Python with mpi4py

Parallel Summation using MPI in Python with mpi4py

Comments 35 Comments

Parallel summation involves distributing the task of summing a large set of numbers across multiple processors or computing nodes, enabling simultaneous computation and aggregation of partial results. Each processor handles a portion of the data, performs local summation, and then communicates its partial sum to a designated root processor. The root processor collects and combines these partial sums to compute the global sum, thereby leveraging parallelism to accelerate the computation process and efficiently handle large-scale data sets. In this tutorial,…

Read More Read More

Parallel Programming Languages and Tools: MPI, OpenMPI, OpenMP, CUDA, TBB

Parallel Programming Languages and Tools: MPI, OpenMPI, OpenMP, CUDA, TBB

Comments 47 Comments

In the age of ever-growing devices, massive data and complex computations, the power of multiple processors simultaneously has become crucial. Parallel programming languages and frameworks provide the tools to break down problems into smaller tasks and execute them concurrently, significantly boosting performance. This guide introduces some of the most popular options: MPI, OpenMPI, CUDA, TBB, and Apache Spark. We’ll explore their unique strengths, delve into learning resources, and equip you to tackle the exciting world of parallel programming. Message Passing…

Read More Read More

MPI: Concurrent File I/O for by Multiple Processes

MPI: Concurrent File I/O for by Multiple Processes

Comments 59 Comments

In this tutorial, we’ll explore an MPI (Message Passing Interface) program using mpi4py to demonstrate how multiple processors can collectively write to and read from a shared file. The detailed tutorial of MPI with a python can be visited here. Code Code Explanation Imports the necessary MPI module from mpi4py which provides bindings for MPI functionality in Python. Initializes MPI communication (comm) for all processes (MPI.COMM_WORLD). rank is assigned the unique identifier (rank) of the current process, and size represents…

Read More Read More

MPI Gather Function in Python

MPI Gather Function in Python

Comments 39 Comments

The gather function is used to gather data from multiple processes into a single process. We’ll go through the provided code, line by line, and understand how the gather function works. The detailed tutorial of MPI with a python can be visited here. Code Explanation This line imports the MPI functionality from the mpi4py library. These lines initialize the MPI communicator (comm) and obtain the total number of processes (size) and the rank of the current process (rank). Each process…

Read More Read More

MPI with Python: Calculating Squares of Array Elements Using Multiple Processors

MPI with Python: Calculating Squares of Array Elements Using Multiple Processors

Comments 34 Comments

In this lab tutorial, we will explore how to utilize multiple processors to compute the squares of elements in an array concurrently using the MPI (Message Passing Interface) library in Python, specifically using the mpi4py module. MPI is a widely-used standard for parallel computing in distributed memory systems. We’ll create a master-worker model where the master process distributes tasks to worker processes, each responsible for computing the square of a subset of the array elements. The detailed tutorial of MPI…

Read More Read More

Data Manipulation with MongoDB Aggregation Framework in Python

Data Manipulation with MongoDB Aggregation Framework in Python

Comments 29 Comments

MongoDB Aggregation Framework is a powerful tool that allows for data manipulation and analysis within MongoDB collections. It provides a flexible and efficient way to process and transform data, enabling users to perform complex operations such as grouping, sorting, filtering, and computing aggregate values. In this lab tutorial, we will introduce the concepts of MongoDB Aggregation Framework, provide a detailed explanation of the code, and walk through each line to understand its functionality. Visit the detailed tutorial here. Code Connection…

Read More Read More

Big Data Technologies

Big Data Technologies

Comments 52 Comments

Big Data refers to datasets that are too large and complex for traditional data processing applications to handle efficiently. It is characterized by the 5 Vs: Volume, Velocity, Variety, Veracity, and Value. Volume refers to the vast amount of data generated, Velocity refers to the speed at which data is generated and processed, Variety refers to the different types of data (structured, semi-structured, and unstructured), Veracity refers to the reliability and quality of the data, and Value refers to the…

Read More Read More

Verified by MonsterInsights