![]() Recoverability allows an ingress or egress tool to retry in the event of a failed operation. Chapter 3 contains more details on these data formats.Ĭompression not only helps by reducing the footprint of data at rest, but also has I/O advantages when reading and writing data. This would convert the data into a form that can be split, such as one JSON or XML element per line, or convert it into a format such as Avro. If your source data is in multiline XML or JSON form, for example, you may want to consider a preprocessing step. Often your source data isn’t in a format that’s ideal for processing in tools such as Map-Reduce. The data format transformation process converts one data format into another. Having the ability to aggregate files or data together mitigates this problem and is a feature to consider. In the context of data ingress, this can be useful because moving large quantities of small files into HDFS potentially translates into NameNode memory woes, as well as slow MapReduce execution times. The data aggregation process combines multiple data elements. How well do distributed log collection frameworks deal with data retransmissions? How do you ensure idempotent behavior in a MapReduce job where multiple tasks are inserting into a database in parallel? We’ll examine and answer these questions in this chapter. Alternatively, updates often are idempotent, because they’ll produce the same end result.Īny time data is being written, idempotence should be a consideration, and data ingress and egress in Hadoop are no different. In a relational database, the inserts typically aren’t idempotent, because executing them multiple times doesn’t produce the same resulting database state. Before we dive into the techniques, however, we need to discuss the design elements you should be aware of when working with data movement.Īn idempotent operation produces the same result no matter how many times it’s executed. Moving large quantities of data in and out of Hadoop offers logistical challenges that include consistency guarantees and resource impacts on data sources and destinations. Let’s start things off with a look at key ingress and egress system considerations. If this is the case, feel free to jump directly to the section that provides the details you need. We’ll cover a lot of ground in this chapter, and it’s likely that you’ll have specific types of data you need to work with. ![]() So as not to ignore some of the emerging data systems, you’ll also be introduced to methods that can be employed to move data from HBase and Kafka into Hadoop. ![]() We’ll look at how you can automate the movement of log files with Flume, and how Sqoop can be used to move relational data. Once the low-level tooling is out of the way, we’ll survey higher-level tools that have simplified the process of ferrying data into Hadoop. We’ll start with some simple techniques, such as using the command line and Java for ingress, but we’ll quickly move on to more advanced techniques like using NFS and DistCp.ġ Ingress and egress refer to data movement into and out of a system, respectively. It goes on to look at low-level and high-level tools that can be used to move your data. ![]() This chapter starts by highlighting key data-movement properties, so that as you go through the rest of this chapter you can evaluate the fit of the various tools. In this chapter you’ll first see how data across a broad spectrum of locations and formats can be moved into Hadoop, and then you’ll see how data can be moved out of Hadoop. Welcome to chapter 5, where the goal is to answer these questions and set you on your path to worry-free data movement. How do you get your log data sitting across thousands of hosts into Hadoop? What’s the most efficient way to get your data out of your relational and No/NewSQL systems and into Hadoop? How do you get Lucene indexes generated in Hadoop out to your servers? And how can these processes be automated? Techniques for moving log files and relational and NoSQL data, as well as data in Kafka, in and out of HDFSĭata movement is one of those things that you aren’t likely to think too much about until you’re fully committed to using Hadoop on a project, at which point it becomes this big scary unknown that has to be tackled.Low-level methods for moving data into and out of Hadoop.Understanding key design considerations for data ingress and egress tools.Hadoop in Practice, Second Edition (2015) Part 2. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |