Explore practical perspectives on software delivery, team collaboration, and technology partnerships that help growing businesses make better decisions and build with confidence.
Amazon DynamoDB is a highly efficient, scalable, and low-latency NoSQL database service, ideal for managing large data volumes and high traffic loads.
Database, the center point of each application, plays a crucial role as data is always needed to execute a project and must be stored in some fashion. As applications grow in complexity and scale, the process of storing and managing data becomes more difficult. When it comes to storing and managing data, developers face the challenging decision of choosing between traditional SQL databases or modern NoSQL databases. This comparison—SQL vs NoSQL—highlights their unique strengths, weaknesses, and ideal use cases, making the decision even more daunting. In this blog, we will dive into the art of SQL vs NoSQL, comparing their features, examining their suitability for different scenarios, and understanding the principles that help us choose between the two. For this exploration, we benchmark one popular SQL database, PostgreSQL, and a leading NoSQL database, DynamoDB. What is Postgresql? PostgreSQL is an open source rdbms(relational database management system) which evolved from the ingres project in the university of california. It follows the traditional ACID (Atomicity, Consistency, Isolation, Durability) properties, making it highly reliable and suitable for various applications. What is DynamoDB? DynamoDB is on the other hand one of fastest growing modern NoSQL databases which is fully managed by AWS(Amazon Web Services) and follows the single table design pattern. It is designed to handle high volumes of unstructured and schemaless data and traffic. It automatically scales up or down according to the workload. Design Patterns Postgresql ACID Properties: It follows the basic acid properties which a must have for sql databases. Atomicity ensures either a transaction will commit or not and there is no in between. Consistency ensures data should remain consistent no matter from where the query comes from. Isolation ensures only one person can write into the database in a single point of time and finally, durability ensures committed data will persist throughout the time. Schema Based Design: It uses a structured schema with different tables. Each table has a primary key(unique identifier) and is connected to another table through a foreign key. It has also join operations to combine two tables to fetch data as well as indexing to fetch data quickly. Here is a sample table design in postgresql where CustomerID is defined as primary key in the Customers table and as far as the Orders table is concern OrderId is the primary key and CustomerID is the foreign key which is outlined in red as how the both table are being connected 3. Indexing: In SQL also in Postgresql uses different types of indexing to retrieve the data faster as the data grows and traditional sql databases tend to slow it down as the query uses sequential searches. So indexing comes as a savior as it uses B-tree to speed up the search. 4. Normalization: Organize data into related tables to reduce data redundancy. There are various types of normalization like 1NF, 2NF,3NF. This process involves large tables divided into smaller ones and multiple ones ensuring each table focuses on one topic. For example let us consider a table order table has the following items Order table Order_id Customer_id Order_date Product_id 1 101 12-11-2003 1,2 2 102 12-11-2003 3,4 As we can see the product_id column holds multiple entries which violates the idea of normalization. As normalization states that each column should hold data atomic we need to break down the table into two and introduce order_items table and new table should look like as follows Order table Order_id Customer_id Order_date 1 101 12-11-2003 2 102 12-11-2003 Order items table Item_id Order_id Product_id Quantity 1 1 1 2 2 1 2 3 3 2 3 4 4 2 4 5 Here we saw the basic operation on normalization which breaks the initial two tables into two and this is called 1NF. However the tables can be further normalized and in order to retrieve data we might need join operation 5. Aggregation: SQL aggregation functions are used to perform calculations on multiple rows of a table’s column and return a single value. Common aggregate functions include SUM, which adds up all values in a column; AVG, which calculates the average; COUNT, which counts the number of rows; MAX, which finds the highest value; and MIN, which finds the lowest value. These functions are often used with the GROUP BY clause to group rows sharing a common attribute and apply the aggregate function to each group. For example, SELECT department, COUNT(*) FROM employees GROUP BY department counts employees in each department. 6. OLAP: OLAP (Online Analytical Processing) in SQL is a powerful data analysis approach used for complex querying and reporting. It enables users to perform multidimensional analysis of large datasets stored in data warehouses. OLAP operations include slicing, dicing, drilling down, and pivoting to view data from different perspectives. SQL-based OLAP leverages aggregate functions, groupings, and hierarchical data structures to provide insights into business metrics. For example, a retail company can analyze sales data by product, region, and time. OLAP enhances decision-making by allowing interactive analysis and quick retrieval of aggregated data, making it essential for business intelligence and strategic planning. DynamoDB Single Table Design: Unlike the traditional SQL design DynamoDB follows single table design meaning it stores all types of data in a single table rather than multiple ones. This type of design introduces two new key words one is partition key and another one is sort key. In single table design we can declare each row as an item and if the items are being corresponding to one partition are called item collection. There some keyword which are being used in the single table design as follows Partition Key: In dynamodb partition key defines how data is being distributed among the partitions.It is also referred as hash key as data data in dynamodb is stored and retrieved as hash. Whenever a data is being inserted it generates a hash and based on the hash it stores the data into the corresponding partition and same goes for retrieving data from a particular partition. Sort Key: Sort key is determined as to how a data is being organized in a particular partition. There is always a possibility that items have the same value for the partition key but there must be a different value for the sort key. Primary Key: Primary key uniquely identifies a row. Sometimes the partition key is regarded as the primary key, however it can be the combination of both partition key and sort key. 2. Event Driven Architecture: dynamoDB streams is a feature of dynamoDB which captures any crud operations in a dynamoDB table and records them into streams. It allows any application to track any types of changes and triggering an event. Each event in the stream is called a stream record which contains information about the item affected and the type of modification. For e When a new order is placed in an order stable, a stream record is generated. A Lambda function can be triggered to process the order, update inventory, send notifications, or replicate the order to another data store. 3. Time to Leave(TTL): Time to leave is a feature of dynamoDB which is responsible for automatically deleting items from a table after a particular time. We don’t need to delete data manually or by query like in SQL. This reduces the storage cost by removing unnecessary data which is no longer needed after a particular amount of time which is about 48 hours by default. 4. Access Pattern: One of the most important yet most challenging tasks is to define the access pattern for your dynamoDB. The design solely depends on what kind of query can happen after the application is being live. The design of partition key and sort key depends on the access pattern. Depending on the query the database will first go to the desired partition where the desired data might reside. So we need to define the access pattern first before designing the database 5. Global and Local Secondary Indexes: It enables another layer of flexibility as far as querying is concerned. In LSI one can add another sort key along with the present sort key and partition key. But if one uses an LSI the partition size automatically becomes a 10gb of size. One can use at most 5 LSI for one table design and it comes free of cost meaning aws will not charge you any penny if one uses any kinds of LSI in a table. On the other hand GSI allows you to add another partition key and sort key in a table. What it does is it copies all the data from the table and in that table we can apply the partition key and sort key. AWS allows you to used up to 20 GSI for a particular table but in that case dynamoDB will charge for each write and read operations. 6. Data Duplication: One of the main features of DynamoDB is the ability to duplicate data, which significantly enhances query performance and response times. However, data duplication increases storage costs, but AWS never charges for storage and only charges for write and read operations. However, while data duplication improves efficiency significantly, it also has drawbacks. Every change must be propagated across all duplicates, increasing the number of write operations that AWS charges for. When to Choose PostgreSQL or DynamoDB? Choose Postgresql Complex Queries and Joins are Necessary: Though join operations are sometimes costlier. PostgreSQL excels with complicated queries and joins, allowing you to quickly access and manage relevant data from numerous tables. Here is an example operations of join query SELECT customers.name, SUM(orders.amount) FROM customers JOIN orders ON customers.id = orders.customer_id GROUP BY customers.name; 2. Extensive Analytical and Reporting Needs: If your application needs heavy analytical reports postgresql should be the go to databases as it can handle extensive queries and also reporting data 3. Data Integrity and Consistency : Foreign keys, unique constraints, and check constraints are among PostgreSQL’s several data integrity capabilities, helping in the maintenance of consistent and correct data. CREATE TABLE products ( product_id SERIAL PRIMARY KEY, name VARCHAR(100) NOT NULL, price NUMERIC CHECK (price > 0), stock INT CHECK (stock >= 0) ); Using constraints to enforce data integrity. Choose DynamoDB Scalability and Performance are Top Priorities: DynamoDB’s seamless scalability and flexibility to manage heavy traffic make it ideal for applications with varying or enormous workloads. Faster Response: If one’s access patterns are defined and you designed your database efficiently DynamoDB will always retrieve data faster than SQL regardless of the data volume it will always fetch data faster. Flexible Data Models and Schema-less Design: DynamoDB’s schema-less design enables flexible data modeling, accepting various data structures without specified schema. High Availability and Fault Tolerance: DynamoDB’s multi-region, multi-active architecture provides high availability and fault tolerance, ensuring that your application remains available even in the face of regional outages. Scenario Restaurant Management System Consider a restaurant management system: customers go through the menus and and according to the menu they place orders and waiters serve the orders.In that case the following requirements are necessary Requirements: Complex queries and joins for reporting. High data integrity and consistency. Scalability for handling variable traffic. Low-latency read and write operations. Probable Access Pattern: Get all the menu Add an order Daily/monthly/yearly total order quantity/amount Recommendation: Postgresql: It can be used for complex queries and joins operations for reporting purposes. Also any type of OLAP operations like how many orders for a particular time or over a period of time or how much sales generates over a period of time. DynamoDB: It can be used for faster read and write operations, for example retrieving all the menu items from the menu or adding an order. As these operations need faster response and also it can be used for variable traffic like when more users are entering the menu item or add in an order it should automatically scale up or down depending on the situations. Conclusion PostgreSQL and DynamoDB are both powerful database systems, with unique advantages based on your application’s requirements. PostgreSQL excels in handling complicated queries, transactions, and relational data models, making it ideal for traditional corporate applications and data warehousing. DynamoDB, on the other hand, provides seamless scalability, faster query response, low latency performance, and flexible data modeling, making it perfect for contemporary, high-traffic applications, IoT systems, and real-time analytics. By evaluating your application’s specific needs around data structure, scalability, performance, and integration requirements with other AWS services, you can make an informed decision between PostgreSQL and DynamoDB. Both databases offer robust solutions to effectively manage and scale your data infrastructure, ensuring optimal performance and reliability for your application.
Amazon DynamoDB is a fully managed NoSQL database service known for its high performance and scalability. One of the critical components that ensure its efficiency is the partitioning mechanism. Understanding how partitioning in DynamoDB data is crucial for designing scalable and high-performing applications. In this blog, we’ll explore DynamoDB partitioning in depth, focusing on its concepts, how it works under the hood, and best practices for optimal performance. Scaling Applications with Proper Partitioning By employing effective partitioning strategies, developers can maximize DynamoDB’s scalability and performance capabilities. For example, an online game might start with only a few users and a small amount of data. However, if the game becomes popular, it can quickly exceed the capacity of the underlying database system. Web-based applications often have hundreds, thousands, or even millions of users at the same time, generating terabytes of new data daily. Databases for such applications need to manage tens of thousands or even hundreds of thousands of read and write operations per second. Amazon DynamoDB is ideal for these types of workloads. As a developer, you can begin with minimal resources and scale up as your application grows in popularity. DynamoDB smoothly handles increasing amounts of data and users. Selecting the proper partition keys at the beginning is a must for seamless scaling. A real-life example is Zoom. In early 2020, when their usage surged from 10 million to 300 million daily meeting participants, DynamoDB scaled effortlessly with just a click and continued to deliver consistent performance. However, to benefit from DynamoDB’s capabilities, you need to plan for the future when designing your table and set the partition key carefully. Otherwise, performance might suffer, and costs could rise due to the need for multiple requests and filtering to find specific data. Understanding how DynamoDB partitions data is crucial for designing scalable and high-performing applications. Concept of Partitioning The main challenge of DynamoDB is the concept of partitioning. It’s just about organizing data. In real life, when we organize something, we keep similar items together. We have to try to keep data like that as much as possible. But if we keep too many similar items together, it will be difficult to find a specific item. So, we need to try to keep the similar items in small partitions. It’s like if all the items in our house are put in one place. First, we separate the kitchen items, drawing room items, and study room items. Now, if someone asks you to bring a salt container, you know you’ll find it among the kitchen items. But it will still take time because all the kitchen items are together and you will need some time to find it among lots of items. So, if we can divide the kitchen items into smaller groups (partition) like utensils, pots and pans, spices, vegetables, cleaning kits, forks, spoons, knives, etc., then if someone asks for salt, you can easily find it among the spices. Similarly, in DynamoDB, we have to keep all the data of a large system in a single table. For good performance (quick data retrieval), we have to organize the full data by partitioning it in the most effective way. So, we have a hypothetical idea of how partitioning works, but to actually apply this idea and maintain proper partitioning for table design, we need to know some theoretical concepts. These are Some Theoretical Concepts Primary Key: This is the main identifier for each item in the table. It ensures that each item can be uniquely identified. There are two types of primary keys: Partition key and Partition key + Sort key. Partition Key: This is part of the primary key and is used to determine the partition in which the data will be stored. It helps in distributing data across different partitions. DynamoDB uses the partition key’s value as input to an internal hash function. The output from the hash function determines the partition (physical storage internal to DynamoDB) in which the item will be stored. Sort Key: This is the second part of the primary key (if used) and allows for sorting within a partition. It helps in organizing related items within the same partition. Data Access Patterns: Understanding how your application will access the data is crucial. This helps in designing the table and partitions in a way that optimizes these access patterns. Hot Partitioning: Avoiding the creation of partitions that get too much traffic compared to others, as this can lead to performance issues. Distributing traffic evenly across partitions is important. Techniques for Choosing the Right Partition Key To choose a partition key, you first need to understand the requirements of your system and its access patterns. It’s not enough to know the access patterns at the start of the project; you also need to consider what types of access patterns might be needed in the future and which ones might be called frequently. Defining the partition key depends on all these factors. Avoid Storing Too Much Data in a Single Partition A DynamoDB query can return a maximum of 1 MB of data in a single operation. This limit applies to both the amount of data returned and the number of items returned. So, if you store too much data in a single partition, you won’t be able to get the desired result in one query, and you’ll need to perform multiple queries. Additionally, filtering a large amount of data can be very costly. Choose a Key that Spreads the Data Evenly In a DynamoDB table, there is no upper limit on the number of distinct sort key values per partition key value. If you needed to store many billions of items in the table, DynamoDB would allocate enough storage to handle this requirement automatically. Consider Access Patterns Understand how your application accesses data. Choose a partition key that aligns with your most frequent query patterns to minimize the need for expensive scans or queries across multiple partitions. When designing your partition key, consider potential future access patterns. For instance, an initial online hotel booking system may start with only a few reservation requests. However, as global customer bookings increase, daily data entries could surge into the hundreds of thousands. Storing all bookings in a single partition initially could become a performance bottleneck as your system scales Don’t use mutable Data in partition key: The primary key in DynamoDB cannot be changed. Using mutable data such as booking statuses (Requested, Pending, Confirmed) in the partition key leads to deleting and rewriting items in the table instead of simple modifications, which is a costly operation. Therefore, it’s essential to avoid using mutable data in the partition key to ensure efficient and cost-effective operations. Understanding Partitioning Through System Example Imagine an online booking platform that handles various types of data related to hotel and flight bookings. The system must manage hotel data, customer profiles, booking transactions, booking history, admin data, and merchant data. The goal is to design a DynamoDB table that can store all this information efficiently, support a wide range of queries, and scale seamlessly as the data volume and user base grow. Types of Data in the System: Hotel Booking: Information about bookings made by customers for hotels. Hotel Data: Details about the hotels listed on the platform. Customer Data: Profiles and personal information of the customers. Flight Booking: Information about flight bookings made by customers. Booking History: Historical records of all bookings made by customers. Admin Data: Profiles and roles of the administrative users managing the platform. Merchant Data: Information about merchants and suppliers associated with the platform. Table Overview with Probable Partition Key: In this example, it is shown how to store data using a partition key in a single DynamoDB table for a complex system and retrieve it quickly. You can see that there are various types of data here. If it were a traditional SQL table, you would need to use numerous tables and joins to query different types of data. For instance, to get a user’s booking data, you would need to join the booking table and the user table. However, here, with a single query, we can retrieve both the customer’s data and their booking information. Proper partitioning has been used to store hotel data, customer data, and booking data, ensuring that even when many customers make bookings simultaneously, the data is stored in separate partitions, and DynamoDB can still retrieve it within single-digit milliseconds. Example Access Points: Hotel Booking: Partition Key (PK) = CUSTOMER#123, Sort Key (SK) = BOOKING#456 Efficiently retrieves all hotel bookings for customer 123: PK = ‘CUSTOMER#123’ AND begins_with(SK, ‘BOOKING#’) Hotel Data: Partition Key (PK) = HOTEL#789, Sort Key (SK) = LOCATION#NYC Retrieves details for hotel 789 located in NYC: PK = ‘HOTEL#789’ AND SK = ‘ LOCATION#NYC’ Customer Data: Partition Key (PK) = CUSTOMER#123, Sort Key (SK) = PROFILE Retrieves profile information for customer 123: PK = ‘CUSTOMER#123’ AND SK = ‘PROFILE’ Partition Key Strategies to Support Easy Expansion By selecting a proper partition key at the outset, you create a scalable, flexible, and efficient data model that supports the seamless addition of new products or features. Adding a new product involves significant schema changes, complex joins, and potential downtime for migrations. Adding a new product is straightforward with no need for schema changes or downtime. You can efficiently store and query new entities using flexible partition and sort keys, ensuring scalability and performance. Adding a New Product (Tour) to the previous system example To add a new product, such as “Tour”, the existing partition key design supports the extension without needing a major overhaul How the Initial Partition Key Design Helps Consistency: Uniform Structure: The existing structure (using composite keys) accommodates new entity types seamlessly. For example, adding TOUR# entries align with existing data access patterns. Example Query: To fetch all tour bookings for a customer Copy Scalability: Horizontal Scaling: DynamoDB automatically allocates partitions as new data types are added, ensuring consistent performance. Example: Adding a large number of tour bookings distributes evenly across partitions. Flexibility: Adapting New Entities: Easily add new types of entities like tours without needing to modify the schema significantly. Example: Store additional attributes for tours within the same table, e.g., Copy Performance: Optimized Queries: New entities can be queried using the same efficient methods designed for existing data. Example: Fetch all information related to a specific tour: Copy By following the best practices for partition keys and leveraging DynamoDB’s flexible schema, you can easily extend your application to include new products and features without the overhead and complexity of traditional SQL databases. Benefits of Proper Partitioning and Access Points Scalability: Proper partitioning allows DynamoDB to scale horizontally by distributing data across multiple partitions. This supports growing data volumes and varying access patterns. Performance: Efficient access points (PK and SK combinations) enable fast and predictable query performance. By selecting appropriate keys, you optimize data retrieval operations. Cost Efficiency: Well-designed partition keys prevent hot partitions and ensure balanced data distribution. This optimizes DynamoDB’s capacity usage and helps manage costs effectively. Conclusion We all know DynamoDB offers high performance and scalability. However, high performance can only be achieved through efficient design. Understanding how DynamoDB partitions data is crucial for designing scalable and high-performing applications. Proper partitioning is the key to get the fullest of DynamoDB. When designing a single table, selecting an appropriate partition key is paramount. It should distribute workload evenly and support your application’s access patterns. Carefully consider how your data will be queried to choose the best partition key.
Amazon DynamoDB is a widely used NoSQL database service that is fully managed. It’s known for being easy to scale, high performance, and low maintenance. This makes it a good option for developers who want to build large applications without dealing with complex database infrastructure. However, even with its advantages, DynamoDB might not be the best choice for every project or organization. As software engineers, choosing the right database is crucial. The database you pick can greatly influence how well your project performs, scales, and succeeds. In this article, we’ll discuss the strengths and weaknesses of DynamoDB, compare it to traditional relational databases (RDBMS) performance, and guide you on When to use DynamoDB—and when it might not be. Understanding DynamoDB An Overview: Before making any decisions, let’s take a quick look at what DynamoDB offers and key features. DynamoDB is a NoSQL database service provided by Amazon Web Services (AWS). It’s built to handle applications that need to respond quickly, even when dealing with large amounts of data. It uses a flexible data model, allowing you to store and retrieve data in various formats, like key-value pairs and documents. Here are some of its key features: Fully Managed Service DynamoDB is a fully managed database service. AWS does all the hard work for you. They take care of setting up hardware, updating software, and making backups. This means you can spend your time building your app instead of managing the database. AWS handles everything behind the scenes, including hardware configuration, replication, and scaling. Seamless Scalability One of the main strengths of DynamoDB is its ability to scale up or down automatically. It can handle large amounts of data and traffic without requiring you to manually adjust settings. This ensures your application performs well, no matter how much data you have or how much traffic you get. Low-Latency Performance DynamoDB is designed to provide fast and consistent performance, with response times in the milliseconds. This makes it ideal for applications that need quick data access, such as real-time data access and quick responses Flexible Data Model DynamoDB is schemaless, so you can store different types of data easily. It supports documents, key-value pairs, and graph data models, allowing you to choose the format that best fits your application’s needs. Durability and Availability DynamoDB automatically copies your data across multiple Availability Zones (AZs) within an AWS region. This ensures that your data is safe and that your application remains available even if one of the data centers goes down. Security DynamoDB integrates with AWS Identity and Access Management (IAM) to give you detailed control over who can access your data. It also uses encryption to protect your information. This means your data is secure both when it’s stored and when it’s being sent somewhere. These features help keep your information private and safe from unauthorized access. Cost-Effective DynamoDB uses a pay-per-use pricing model, which means you only pay for the resources you actually use. It also scales its capacity automatically based on your workload, which can save you money compared to running your own database. Now, let’s explore the scenarios when it’s a good fit and when it might not be the best choice. DynamoDB vs. RDBMS Performance For a better understanding let’s follow the chart below: This image is a performance comparison between MySQL and DynamoDB as the data size increases. Most traditional Relational Database Management Systems (RDBMS) like MySQL, PostgreSQL, Oracle, and SQL Server generally show similar performance patterns as data size increases. Here’s how to interpret the graph: Y-Axis (Performance): The performance of the database system is represented on the vertical axis, ranging from “Blazing fast” at the bottom to “Painful” at the top. X-Axis (Data size): The horizontal axis shows the size of the data, ranging from 1GB to 1TB. RDBMS(MySQL) Performance Curve: The curve for MySQL shows that The performance of MySQL starts off very strong with smaller data sizes, around 1GB, where it’s “Blazing fast.” However, as the data size increases, moving towards 100GB to 1TB, the performance gradually declines. It goes from “Blazing fast” to “Sluggish,” and with even larger data sizes, it can become “Painful” to work with. DynamoDB Performance Line: In contrast, DynamoDB’s performance is shown as a horizontal line that remains steady as the data size increases. staying constant across all data sizes. It indicates that DynamoDB maintains consistent performance regardless of how much the data size increases. The chart above is a rough sketch, but the overall point stands. At certain levels of data and transaction volume, an RDBMS will have faster response times than DynamoDB. So MySQL might beat DynamoDB at the median, but that’s not the full story. In this chart, we see that RDBMS(mySQL) latency will get worse as the amount of data in our database increases, whereas DynamoDB will provide consistent performance as our data increases. This same relationship holds as the number of concurrent requests increases. Hence, Traditional databases perform well with small data volumes but struggle as the data grows, leading to slower retrieval times. DynamoDB offers a solution to this challenge by offering consistent performance no matter how large the data volume gets, but using it efficiently is crucial. However, It’s important to note that while the graph shows an idealized view of DynamoDB’s performance, the reality is a bit more complex. Efficient data retrieval and long-term scalability in DynamoDB rely on proper table design, careful selection of partition keys, and effective use of sort keys. If the database design is flawed, DynamoDB’s performance can degrade over time as data volume increases, much like other database systems. On the other hand, a well-designed table structure can ensure fast data retrieval, even with large amounts of data. These points highlight that while DynamoDB offers great scalability and performance potential, achieving the ideal results shown in the graph requires careful attention to database design principles. The consistent performance line for DynamoDB is possible, but it depends on proper implementation and optimization. Why Amazon DynamoDB Isn’t for Everyone Despite its many advantages, DynamoDB may not be the best choice for every use case. Here are some reasons why you might choose not to use DynamoDB: Complex Querying Requirements DynamoDB is good at simple data lookups but struggles with complex query options. It works best when you know exactly what you’re looking for. However, if you need to combine data from different places, do complicated calculations, full-text search or search through lots of text, DynamoDB might not be the best choice. In these cases, you might want to use a regular traditional relational database or a special search tool instead. These can handle more complex data tasks that DynamoDB can’t do easily. Data Duplication DynamoDB uses a denormalized data model, which can lead to data duplication. If your app needs to keep data neat and tidy, with each piece of information stored only once, a traditional database like MySQL might be a better fit. These databases are designed to manage organized, non-repetitive data effectively. Think of it this way: DynamoDB is like a messy, quick-access filing cabinet that prioritizes speed and convenience. In contrast, a traditional database is more like a carefully organized library system that focuses on orderliness and organization. Your choice between these options should depend on whether speed or precise organization is more important for your needs. Item Size Limits DynamoDB has a limit of 400 KB per item. If your application needs to store or retrieve items larger than this, you may need to look for other solutions or create workarounds, which can add extra complexity. Consistency Needs DynamoDB offers strong consistency for reads, but for global tables, it uses an eventual consistency model. This might not be suitable for applications that need immediate consistency across all operations. Strict Schema Requirements If your application has rigid schema requirements or needs enforced data integrity constraints at the database level, an RDBMS might be a better fit. Complex Transactions DynamoDB can handle ACID transactions but has limits on how many items you can include at once. If your application needs to handle complex transactions across multiple tables, a traditional RDBMS might be a better fit. SQL-Based Tools and Expertise If your team is experienced with SQL and relies heavily on SQL-based tools, switching to DynamoDB might be difficult. It may require a steep learning curve and significant changes to your current tools and workflows. Vendor Lock-in Using DynamoDB ties you to the AWS ecosystem. If you need to migrate to another database system in the future, it could be challenging due to DynamoDB’s unique data model and query language. Cost for High-Volume Workloads If your application deals with a large amount of data on a regular basis, DynamoDB’s pay-per-use pricing might become quite expensive. In these situations, using a well-optimized relational database on dedicated servers could be a more cost-effective option. These points highlight why DynamoDB might not be the right choice for every application. It’s important to carefully evaluate your needs and consider alternative options if any of these factors are a concern. Scenarios Where DynamoDB is Advantageous Despite its limitations, DynamoDB shines in certain scenarios. Here are some cases where DynamoDB is the preferred choice: Microservices Architectures DynamoDB is a strong choice for applications that use microservices. Its ability to quickly scale and provide low-latency performance makes it ideal. Each microservice can have its own DynamoDB table, making it easy to manage and scale data separately for each part of your application. Real-Time Data Processing DynamoDB is fast, making it ideal for apps that need to process data in real-time. If your app needs to show updated information instantly, like in real-time analytics dashboards, leaderboards, or stock trading platforms, DynamoDB can handle it. High-Scale Applications DynamoDB is perfect for handling large amounts of data and traffic. It can automatically adjust to meet changing needs, so it’s great for applications that grow quickly or face sudden spikes in traffic. This makes it a good choice for social media platforms, gaming apps, and IoT systems with millions of users or devices. Serverless Architectures DynamoDB works well with AWS Lambda and other serverless technologies. This makes it an ideal choice for applications that need to automatically scale up or down based on demand, without the hassle of managing servers. It’s particularly useful for event-driven systems and microservices due to this easy integration. Mobile and Web Applications DynamoDB’s flexible schema is perfect for mobile and web apps that evolve quickly. Since it is schema-less, you can easily update your data structure without complicated migrations. This makes it much simpler to adapt to changing needs and new features. Global Applications DynamoDB’s global tables feature lets you replicate data across multiple AWS regions. This means users all over the world can access your data quickly, with low latency, no matter where they are. It’s ideal for applications that need fast, reliable access to data on a global scale. IoT Data Storage DynamoDB is great for apps that collect data from IoT devices because it can easily scale and handle fast data writes. Content Management Systems DynamoDB’s support for document models and global secondary indexes makes it a good choice for content-heavy apps that need fast read access and flexible data structures. Situations Where DynamoDB Might Not Be the Best Choice While DynamoDB has many strengths, there are situations where it may not be the best fit: Large Data Objects or BLOB Storage DynamoDB has a maximum item(row) size limit of 400 KB. If your application needs to store large objects or BLOB (Binary Large Object) data, you’ll need to use external storage solutions like Amazon S3 and store references in DynamoDB. This adds complexity to your architecture and may not be ideal for all use cases. Alternatively, databases supporting larger item sizes might be more suitable. Pagination Limitations DynamoDB has some limitations when it comes to handling paginated data, which might not work well for certain applications: Limit on Data per Query DynamoDB can only return up to 1 MB of data per query. If your query results exceed this limit, you’ll need to break the results into smaller chunks and request them in multiple parts. This can make it more complex to manage large sets of data. No True Offset-Based Pagination Unlike traditional SQL databases, DynamoDB doesn’t support true offset-based pagination. Instead, it uses a “LastEvaluatedKey” to continue from where the previous query left off. This can make it challenging to implement features like “skip to page X” or display the total number of pages. Inconsistent Page Sizes DynamoDB’s pagination is based on data size, not the number of items. This means that the number of items on each “page” can vary, especially if your items are different sizes. This inconsistency can be confusing for users who expect each page to show the same number of items. Performance Issues with Deep Pagination As you paginate through large datasets, performance can degrade because DynamoDB needs to scan through all preceding items to reach the desired page. This can be particularly problematic for applications that need to access data deep within large result sets. For applications with these requirements, a traditional RDBMS or a database with more robust pagination support might be a better choice. Frequent Schema Changes While DynamoDB’s flexible schema is often an advantage, it can become a challenge if your application requires frequent and substantial schema changes. Managing these changes in a NoSQL environment can be complex and error-prone. Legacy Application Migration Moving old applications from relational databases to DynamoDB can be very difficult. It often requires big changes to how your data is organized and how your application functions. You’ll need to carefully adjust both your data model and application code logic to fit DynamoDB’s structure which can be time-consuming and complex. Applications with Complex Relationships DynamoDB’s querying features are less advanced compared to relational databases. If your application needs to perform complex queries, such as joining data from multiple tables, DynamoDB might not be the best fit. It works better for simpler queries and more straightforward data retrieval. ACID Transactions Across Multiple Items DynamoDB supports ACID transactions, but only within a single partition key. If your application needs to handle complex transactions that involve multiple items or tables, a relational database is often a better choice. It can manage these types of transactions more effectively across different tables. Strong Consistency Requirements DynamoDB offers strong consistency,but it comes at the cost of higher latency and read capacity. If your application needs immediate and consistent data for every read, DynamoDB might not be ideal. For complex transactions or high consistency needs, like those in financial systems, DynamoDB’s eventual consistency and limited transaction support might not be sufficient. Cost-Sensitive Operations with Predictable Workloads If your application has steady and predictable usage patterns, DynamoDB’s pay-per-request pricing could end up being more costly. In such cases, a traditional relational database might be more budget-friendly if you don’t need DynamoDB’s extra scalability and flexibility. Small-scale Applications with Simple Requirements For small applications with simple data storage needs and low traffic, DynamoDB might be more complex than necessary. The effort to learn and set up DynamoDB could outweigh its benefits. In such cases, a simpler solution might be more practical and cost-effective. Conclusion Amazon DynamoDB is a powerful and flexible NoSQL database service that works well in situations needing high scalability, fast performance, and flexible data models. Its strengths make it an excellent choice for many modern applications, particularly those dealing with large-scale, real-time data processing and serverless architectures. However, DynamoDB is not a universal solution. Its limitations in complex querying, strict consistency, and transaction support make it less suitable for applications with complex data relationships or those requiring advanced SQL-like operations. The decision to use DynamoDB should be based on a thorough analysis of your application’s requirements, including scalability needs, data model complexity, query patterns, and consistency requirements. It’s also crucial to consider factors such as your team’s expertise, long-term costs, and potential vendor lock-in. In many cases, a hybrid approach combining DynamoDB with other database technologies might provide the best of both worlds. For example, you could use DynamoDB for high-throughput, low-latency operations while employing an RDBMS or a search engine for complex queries and analytics. Ultimately, the key to success lies in understanding the strengths and limitations of DynamoDB and aligning them with your specific use case. By carefully evaluating your needs and considering the factors discussed in this blog, you can make an informed decision on whether DynamoDB is the right choice for your project. Remember, the database landscape is constantly evolving, and what works best today might change in the future. Stay informed about new features and improvements in DynamoDB and other database technologies to ensure your choices remain optimal for your applications’ growing needs.
Effectively manage the change of field data type of PynamoDB, an object-relational mapper (ORM) for DynamoDB. Ensure smooth database schema updates and maintain application performance and integrity with this guide. PynamoDB, with a Python interface, is a great ORM for AWS DynamoDB. DynamoDB is a NoSQL database with scalability and security and it has awesome API and SDKs. PynamoDB wraps the underneath APIs of DynamoDB and gives a beautiful pythonic way to use DynamoDB. In real-world our DB fields are not constant. It can change with the requirements update of our project. The hard part is when you have to maintain the old and new features. For example, a field data type can change from String to Binary. So if any API e.g. REST API with versioning is working with DynamoDB, you have to accept both old data type (String) and new data type (Binary). We are going to find a solution in such a case. Setting the goal Let’s start with setting a goal. Assume we have a field with Dynamo data type List. But after times it changes to Map. So we will do Accept Map object and save to DynamoDB Accept List object and change it to Map according to requirement and save to DynamoDB as our field changed to Map Return Map object according to requirement change Setting the project Let’s add a couple of files to do our task. # Add project folder mkdir change-field-ddb cd change-field-ddb # Add Pipfile for requirements mange with pipenv touch Pipfile # For Custom PynamoDB attributes touch attributes.py # Add models.py to add pynamo model touch models.py # To test everything touch test.py Add python requirements Add following requirements to Pipfile [dev-packages] pipenv = "==2020.8.13" moto = "==1.3.14" [packages] pynamodb = "==4.3.2" [requires] python_version = "3.8" Now initiate pipenv with pipenv install Now the project will look like tree . ├── attributes.py ├── models.py ├── Pipfile ├── Pipfile.lock └── test.py The changed datatype Let’s assume our previous datatype (List) for a field result is [3.75, 3.17, 3.90, 3.67, .......] So the result stores CGPA of a student from 1st semester to final semester e.g. result[0] as 1st semester, result[1] as 2nd semester, and so on. But we will make it better (Map) at the updated version { 'semester 1': 3.75, 'semester 2': 3.17, 'semester 3': 3.90, 'semester 4': 3.67, .................. } PynamoDB attributes Add the custom PynamoDB attribute to accept old (List) and new (Map) attribute and saves only as Map. But at return time it will return old (List) data as Map and also new data as Map. Also, we are going to use custom UUIDAttribute which will be used as a hash key. Copy The PynamoDB Model We are going to start with a simple PynamoDB Model only with 2 fields Copy Time to Test Add some tests in test.py Copy Run tests with pipenv shell python3 test.py Full code can be found here: https://github.com/melon-ruet/change-field-ddb
Introduction When we first start working with a third-party API, the version we started with may become obsolete after a few years since the legacy API may no longer be supported by the original creators or contains security problems when we need to migrate to newer versions. There could be a variety of other reasons to upgrade, such as the expense of maintaining the legacy API being greater and inefficient than the latest version, as well as support for new features or support from the original providers. In this article, we will be working on Instagram Private API (legacy API) and Instagram Graph API (new API). The Instagram Graph API, which is a set of Facebook Graph API endpoints that let developers access information from Instagram Professional accounts introduced in January 2021. Before it, one of our team at Craftsmen used to use the Instagram Private API to publish social media content on Instagram. So, before we start, let’s call Instagram Graph API as IG-API and Instagram Private API as IP-API for this entire blog, and let’s first explain why we were encouraged to switch from IP-API to IG-API and address an approach for working with both of them. Reasons for switching to new API A few months after the first initial release of IG-API, we had cases where our client’s Instagram account got temporarily blocked or suspended while using our services. We investigated what might be the prior reason behind it and got multiple open issues in the IP-API GitHub repository. From our research, we came to the conclusion that IP-API seems to work in that it tries to create a device instance and makes Instagram think that the request is coming from an actual physical device. But, if we keep doing it on every post publish, and try to use any kind of web service like AWS Lambda to handle the task. Instagram might get suspicious about the incoming requests as we have no control over the IP of the AWS Lambda which it is triggering from. Think of it in this manner, you might get blocked if you keep changing your IP address using VPN and keep posting images from different regions with different IP addresses, you might get locked out or suspended for security reasons. Our workflow for both legacy and new API We started developing support for IG-API but unfortunately, of all the features which were offered by IP-API only two were offered by the IG-API. So, we have to make the systems work aside from each other in the following manner. While implementing the IG-API, we came across a unique problem, as the new IG-API does not directly support fetching post information that IP-API made. So how did we fix this problem? Let’s start small, first, whenever we publish a post on Instagram using IG-API we get a post id for the post which is later used to fetch information about the post using the graph endpoints. Now, though a similar post id is also expected from the IP-API as well unfortunately it sends us a combination of post ig_id and account ig_id connected with an underscore and no post id exists in the IP-API response which is needed to fetch post information with IG-API. We can fetch a user’s list of media post id with ig_id with IG-API for a specific time range which enables us to narrow down the response. As we keep the logs for each lambda execution’s start and end time for every successful post, we have used it to fetch the list of media post id and ig_id to later find the exact match for the IP-API created post. Problem solved, yay!! But what if we were not that much lucky? The above solution was possible as we had the necessary information at our disposal but what if we did not have any time logs or IG-API does not support the time-based pagination? An alternative approach So for the not-lucky case, one might suggest, why not do a system-wide database migration with the new IG-API post id. This might be a good idea, but we had to think of our use cases and costs. As we use AWS Dynamodb for our database, scanning and updating the whole database for post information of Instagram Provider seems overkill to solve the problem. Not to mention, it’s expensive and the client does not have any use cases where they regularly need to see the old post information. So, on another approach, we can fetch all the post id with ig_id using IG-API then make a cross-reference using the IP_API post id’s post ig_id and finally migrate the existing database with the new id and send a response. But the latency of the above-mentioned process is going to be high as we are fetching all the post id and ig_id for each request which is redundant and chances are our AWS lambda might get timed out if the number of posts is too high. Let’s modify the solution a little bit more, we can introduce cache storage that saves the post id and ig_id while the registration and later find the exact id from there. The cache storage does not need to be synchronized with the updated post ids as the IP-API is now no longer supported. If it is still to be supported then a scheduler needs to be executed at a certain time interval to fetch and store the updated post ids of IG-API. Hope you have enjoyed reading this article. Thank you for your time.
